Azure Synapse Analytics Blog articles https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/bg-p/AzureSynapseAnalyticsBlog Azure Synapse Analytics Blog articles Wed, 17 Aug 2022 07:59:13 GMT AzureSynapseAnalyticsBlog 2022-08-17T07:59:13Z Updated Synapse Analyzer Report: Workload Management and ability to monitor large data warehouses https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/updated-synapse-analyzer-report-workload-management-and-ability/ba-p/3580269 <P>In May 2022 we <A title="Azure Synapse Analyzer Report to monitor and improve Azure Synapse SQL Dedicated Pool performance" href="https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analyzer-report-to-monitor-and-improve-azure/ba-p/3276960" target="_self">announced</A> the first version of Synapse Analyzer Report that will help you with table, index, and statistics information and report issues on these topics.</P> <P>&nbsp;</P> <P>With help from Nick Salch, Fretz Nusan, and Synapse CSE team, today we have an <A href="#" target="_self">update (version 2.0) to the Synapse Analyzer report</A>:</P> <UL> <LI>The addition of the Workload Management Report page to the existing report</LI> <LI>A new separate CTAS version of Synapse Analyzer Report for extra-large data warehouses</LI> <LI>Fixed below error during refresh in Power BI Desktop: <UL class="lia-list-style-type-circle"> <LI><EM>"Query 'Query Name’ (step 'Expanded All Tables') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination."</EM></LI> </UL> </LI> </UL> <H3>New report page - Workload Management Report:</H3> <P>The new version (v2.0) contains additional reports to detect issues in configuration of Workload Management settings in Synapse Dedicated Pool.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Workload Management" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/390241i236FB8708B32568D/image-size/large?v=v2&amp;px=999" role="button" title="Workload-Management.png" alt="Workload Management" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Workload Management</span></span></P> <P>&nbsp;</P> <P>Following is a brief description of the Workload Management report page.</P> <P>&nbsp;</P> <H4>Workload Management:</H4> <P>This page provides information on workload management and related settings in Synapse dedicated SQL pool. It highlights some of the common warnings admins should be aware of when setting the resource and request percentage for effective workload governance. Please review issues and warnings on this page to ensure you are using the optimal strategy for your workload groups and classifiers. At any time you can click on the "?" icon on the visual to view more details on guidance.</P> <H5>&nbsp;</H5> <H5>WLM Issue Summary Table:</H5> <P>This table provides summary of all issues detected in your WLM environment. You can hover on each issue to find a brief description and/or click on an issue to find groups affected by the respective issue. You can also click on the URL link to find more information about workload management.</P> <H5>&nbsp;</H5> <H5>Group Details Table:</H5> <P>The table provides details for each workload group. Groups highlighted suggest that one or more issues for that group. You can hover on the groups flagged to find issues identified for that group.</P> <UL class="lia-list-style-type-circle"> <LI>Flag for effective values suggests there is mismatch between config and effective value at request or resource level.</LI> <LI>Flag on request min config suggests mismatch in request min vs request max values.</LI> <LI>Flag on resource min config suggests deviations in resource min configurations.</LI> </UL> <P>&nbsp;</P> <H5>Requests/Timeouts Summary:</H5> <P>The table provides snapshot of the requests and timeouts summary at the time when data was last refreshed/updated.</P> <P>&nbsp;</P> <P>The table below provides details on issues that have been added for workload management.</P> <P>&nbsp;</P> <TABLE border="1" width="100%"> <TBODY> <TR> <TD width="20.503597122302153%" height="33px" class="lia-align-left"><FONT size="4"><STRONG>Issue </STRONG></FONT></TD> <TD width="79.49640287769785%" height="33px" class="lia-align-left"><FONT size="4"><STRONG>Issue Description</STRONG></FONT></TD> </TR> <TR> <TD width="20.503597122302153%" height="144px"><STRONG>Importance Mismatch</STRONG></TD> <TD width="79.49640287769785%" height="144px">Mismatch in workload group importance and workload classifier importance for a workload group. Importance can be set separately at the workload group or workload classifier level. Whichever importance is higher is the one that will be assigned to the query. This test is just to make the user is aware that there are two different levels of importance set to make sure there is no confusion on the classified query's importance. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="147px"><STRONG>Request Max Mismatch</STRONG></TD> <TD width="79.49640287769785%" height="147px">Mismatch in configured and effective values of max request grant percentage for a workload group. The system will allow you to set any valid values for your request max and this will be your configured or desired value. The effective value is the actual value that is being used based on the configuration of other workload groups in the system. If your request max is set to 100%, but there is another workload group with a 10% minimum, then your effective request max will be lowered to 90%.&nbsp;Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="30px"><STRONG>Request Min Mismatch</STRONG></TD> <TD width="79.49640287769785%" height="30px">Mismatch in configured and effective values of min request grant percentage for a workload group. The system will allow you to set any valid values for your request minimum and this will be your configured or desired value. The effective value is the actual value that is being used based on the configuration of other workload groups in the system. If your request min is set to 50%, but there are other workload groups with combined minimums totaling 60%, then your effective request min will be lowered to 40%. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="30px"><STRONG>Resource Max Mismatch</STRONG></TD> <TD width="79.49640287769785%" height="30px">Mismatch in configured and effective values of cap resource percentage for a workload group. The system will allow you to set any valid values for your resource max (workload group max) and this will be your configured or desired value. The effective value is the actual value that is being used based on the configuration of other workload groups in the system. If your resource max is set to 100%, but there is another workload group with a 10% minimum, then your effective resource max will be lowered to 90%. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="30px"><STRONG>Resource Min Mismatch</STRONG></TD> <TD width="79.49640287769785%" height="30px">Mismatch in configured and effective values of min resource percentage for a workload group. The system will allow you to set any valid values for your resource minimum and this will be your configured or desired value. The effective value is the actual value that is being used based on the configuration of other workload groups in the system. If your resource minimum is set to 50%, but there are other workload groups with combined minimums totaling 60%, then your effective request min will be lowered to 40%. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="30px"><STRONG>Resource Min NonZero</STRONG></TD> <TD width="79.49640287769785%" height="30px">Resource min percentage is nonzero for a workload group. You can set a workload group to have a minimum percentage allocated at all times, which is true regardless of whether or not there are queries running in this group. This test is to notify the user that there is a group that has a permanent resource grant because it will lower the concurrency resources available to all other groups and in many cases lower your overall concurrency. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD width="20.503597122302153%" height="30px"><STRONG>Resource Min Over10%</STRONG></TD> <TD width="79.49640287769785%" height="30px">Resource min percentage is &gt;10% for a workload group. When you set a workload group minimum it is allocated at all times regardless of whether or not there are queries running in this group. If this minimum is set to high, then it can have a dramatic affect on the overall concurrency available to the system since queries not in this group can never tap into this granted percentage. There may be rare scenarios where this is a valid configuration such as a fairly constant workload that regularly uses more than this minimum, but you are more likely to achieve higher concurrency by removing or lowering this minimum, capping certain workloads, and allowing the shared resources to be dynamically allocated based on importance. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> <TR> <TD><STRONG>Request Min vs Request Max mismatch</STRONG></TD> <TD>Request min grant percentage and request max grant percentage do not match for a workload group. On a per-request basis within a workload group you can set a separate minimum and maximum resource grant. If there are enough resources available it will grant the maximum, but if there are not it will lower the allocation until it gets to the minimum at which point if there are still not enough resources the query will queue. However, in most scenarios you do not want to set a separate minimum and maximum request grant because it could result in varying query runtimes based on how large of a grant the query is given at different times. Refer <A title="Workload management with resource classes in Azure Synapse Analytics" href="#" target="_self">Workload Management</A>&nbsp;for details.</TD> </TR> </TBODY> </TABLE> <H3><BR />CTAS version for XL data warehouses:</H3> <P>With this update we are also bringing a parallel copy of Synapse Analyzer report specifically targeted towards scenarios where Synapse Dedicated Pool database has greater than 20K tables and the normal refresh of synapse analyzer report times out after a long period (greater than 5 hours).</P> <P>&nbsp;</P> <P>For this version of the report the Synapse Admin has to run the Create Table script on the dedicated Pool database which is being analyzed. The script will:</P> <OL class="lia-list-style-type-lower-alpha"> <LI>Create new schema named “SynapseAnalyzer” under same database</LI> <LI>Execute Create Table as Select (CTAS) statements to create tables in SynapseAnalyzer schema and prepopulate it with details from the same database.</LI> <LI>These tables are then used as source for the Power BI template file “Synapse best practice analyzer_with_CTAS.pbit”</LI> <LI>The PBI template will then connect to SynapseAnalyzer schema and pull all the information from the tables that are prepopulated in above steps.</LI> </OL> <P><STRONG>Important:</STRONG> You need to run the scripts in given order.</P> <P>&nbsp;</P> <OL> <LI>The “CreateTablesForSynapseAnalyzer_CTAS.sql” script creates new schema named “SynapseAnalyzer” and then creates all required tables as a part of CTAS</LI> <LI>Only after step 1 above is successfully complete you can use “Synapse best practice analyzer_with_CTAS.pbit” for creating report using the PBI template file.</LI> </OL> <P>&nbsp;</P> <H3>Fixed error during desktop refresh:</H3> <P>For many users, running load data through Power BI desktop, the “Load Data” used to result in following error:</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="markdown">All Tables Loading blocked by failures with other queries. ColumnstoreDensity Query 'ColumnstoreDensity' (step 'Expanded All Tables') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination. StatsSummary Loading blocked by failures with other queries. TableSizes Query 'TableSizes' (step 'Expanded All Tables') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination. …</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P>While there is a details section in the Troubleshooting document on the Github location for troubleshooting this error, the overall first experience with the tool was not optimal.&nbsp;With a new version of <A title="Synapse Analyzer Report" href="#" target="_self">Synapse Analyzer Report</A>&nbsp;we have fixed this issue so you should not be getting the above error.</P> <P>&nbsp;</P> <P>In future we plan to continue with more features and reports. Thank you very much for using Synapse Analyzer. As always, we would be happy to receive any feedback you may have on the report.</P> <P>&nbsp;</P> <P>&nbsp;</P> Tue, 16 Aug 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/updated-synapse-analyzer-report-workload-management-and-ability/ba-p/3580269 SanjayRaut76 2022-08-16T15:00:00Z Accelerate your Oracle data warehouse migration to dedicated SQL pools in Azure Synapse Analytics https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/accelerate-your-oracle-data-warehouse-migration-to-dedicated-sql/ba-p/3595213 <P><A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A> is a limitless analytics service that brings together enterprise data warehousing and Big Data Analytics. It gives you the freedom to query data on your terms at scale using <A href="#" target="_blank" rel="noopener">Synapse SQL</A>. <A href="#" target="_blank" rel="noopener">Dedicated SQL pools in Azure Synapse Analytics</A> are powerful massively parallel processing (MPP) distributed query engine which offers enterprise data warehousing capabilities for workloads with predictable performance and cost. This <A href="#" target="_blank" rel="noopener">Massively parallel processing (MPP)</A> engine uses a <A href="#" target="_blank" rel="noopener">node-based architecture</A> to optimize the quick execution of the most complex queries on large volumes of data.</P> <P>&nbsp;</P> <P>Modernizing your on-premises data warehouse by migrating to Azure Synapse reduces maintenance costs, greatly improves performance, and provides high availability. However, migrating from an on-premises data warehouse to a Cloud data warehouse can be complex and time consuming. Extract, transform and load (ETL) processes, large amounts of data and reports built over the years need to be migrated to Azure Synapse quickly while navigating differences in architecture and design, database objects and data types, performance tuning, ETL and SQL.</P> <P>&nbsp;</P> <P>Due to the enhanced security and scalability of compute and storage resources that&nbsp;<A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A> offers along with flexibility and seamless integration; customers implement a change management strategy to migrate to <A href="#" target="_blank" rel="noopener">dedicated SQL pools in Azure Synapse Analytics</A> from their existing on-Premises data warehouse sources. In this blog, we introduce the migration guide which not only resolves the common issues faster but also helps accelerate the migration of Oracle data warehouse to <A href="#" target="_blank" rel="noopener">dedicated SQL pools in Azure Synapse Analytics</A>. Here is what the <A href="#" target="_blank" rel="noopener">oracle migration guide</A> covers:</P> <P>&nbsp;</P> <OL> <LI><STRONG>Design and performance for Oracle migration</STRONG> – We start by describing the database, data types, and database objects that need to be changed to migrate to <A href="#" target="_blank" rel="noopener">dedicated SQL pools in Azure Synapse Analytics</A>. Then we discuss the similarities and differences in performance tuning along with best practices that can be adapted for a highly performant data warehouse along with various ingestion methods supported.</LI> <LI><STRONG>Data, ETL, and load migration considerations</STRONG> – Here we set out the initial decisions that need to be made, and best practices to minimize migration risk. We describe a suggested approach to determining the size of the database and its volume, along with ETL design and tools that can be utilized.</LI> <LI><STRONG>Security access and Operations</STRONG> - Both Oracle and Azure Synapse Analytics implement database access control via a combination of users, roles, and permissions. Both use standard SQL and therefore it may be possible to automate the migration of existing user ids, roles, and permissions. With minimal risk and user impact, most Oracle operational tasks can be implemented in <A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A>. This section contains how all security access and operations can be easily migrated from Oracle.</LI> <LI><STRONG>Visualization and reporting for Oracle migrations</STRONG> – This section contains the considerations and approach to analyze and migrate business intelligence dashboards and reports.</LI> </OL> <OL start="5"> <LI><STRONG>Minimizing SQL issues</STRONG> - There are several differences in Structured Query Language (SQL) support between Oracle and Azure Synapse Analytics, including data definition language (DDL) and data manipulation language (DML). This section contains the most common approaches to bridge this gap along with guidelines to convert Oracle built-in SQL functions to <A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A>.</LI> <LI><STRONG>Microsoft and 3<SUP>rd</SUP> party tools</STRONG> – <A href="#" target="_blank" rel="noopener">SQL Server Migration Assistant</A> (SSMA) for Oracle not only automates code translations for small to medium sized workloads but also helps adapt code between Oracle and <A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A>. This section contains the details of all the Microsoft as well as the products that <A href="#" target="_blank" rel="noopener">Microsoft partners</A> offer for migrations.</LI> <LI><STRONG>Implementing modern data warehouses</STRONG> – The On-Premises data warehouse when migrated to <A href="#" target="_blank" rel="noopener">dedicated SQL pools in Azure Synapse Analytics</A> can be integrated seamlessly with Microsoft’s Azure analytical ecosystem. The migrated data warehouse can be modernized by taking advantage of Microsoft technologies such as <A href="#" target="_blank" rel="noopener">Azure Data Lake Storage</A> for ingestion and cost-effective storage, <A href="#" target="_blank" rel="noopener">Azure Data Factory </A>for self-service data integration and <A href="#" target="_blank" rel="noopener">Common Data Model </A>to share consistent trusted data across multiple technologies. Also, Microsoft’s data science technologies and <A href="#" target="_blank" rel="noopener">Azure HDInsight</A> can be leveraged to process massive amounts of data in a cost-effective manner and to predict outcomes using <A href="#" target="_blank" rel="noopener">Azure Machine Learning</A>. <A href="#" target="_blank" rel="noopener">Azure Event Hubs</A>, <A href="#" target="_blank" rel="noopener">Azure Stream Analytics </A>and <A href="#" target="_blank" rel="noopener">Apache Kafka</A> help to integrate streaming data. All the Microsoft technologies when combined not only unlocks the potential to derive past, present, and future insights but also helps business discover more potential data sources and make data-driven informed decisions thereby helping businesses flourish to greater heights.</LI> </OL> <P>&nbsp;</P> <P>We’re happy to share the new <A href="#" target="_blank" rel="noopener">migration guide</A>&nbsp;to <A href="#" target="_blank" rel="noopener">dedicated SQL pools in Azure Synapse Analytics</A> from Oracle. It provides steps, process and guidelines on migrations and insights on the capabilities of <A href="#" target="_blank" rel="noopener">SQL Server Migration Assistant</A> (SSMA) for Oracle, which helps automate the migration. Take advantage of the migration guide to modernize your data warehouse and accelerate your business by integrating with the <A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A> ecosystem.</P> <P>&nbsp;</P> <P>Check out our <A href="#" target="_blank" rel="noopener">Migration Guides</A> for Teradata and IBM Netezza migrations.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> Thu, 11 Aug 2022 23:00:51 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/accelerate-your-oracle-data-warehouse-migration-to-dedicated-sql/ba-p/3595213 ajagadish 2022-08-11T23:00:51Z Exciting new release of SynapseML https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/exciting-new-release-of-synapseml/ba-p/3589606 <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="NellieGustafsson_0-1659485604926.png" style="width: 607px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/392961i25DC2D7D3C5DE052/image-dimensions/607x319?v=v2" width="607" height="319" role="button" title="NellieGustafsson_0-1659485604926.png" alt="NellieGustafsson_0-1659485604926.png" /></span></P> <P>&nbsp;</P> <P>Today we’re excited to announce another great release of SynapseML v0.10.0 (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines.</P> <P>This is great news for all our SynapseML fans out there and this blog post covers the latest additions. First, a quick recap of what SynapseML is for all the readers who have not yet used the library.</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">SynapseML</A> is a massively scalable (feel free to spin up hundreds of machines!) machine learning library built on Apache Spark. SynapseML makes it easy to train production-ready models to solve problems from simple classification and regression to anomaly detection, translation, image analysis, speech to text, and just about any ML challenge you are facing. &nbsp;Under the hood, SynapseML integrates a wide array of ML technologies such as LightGBM, Vowpal Wabbit, ONNX, and the Cognitive Services into a single easy to use API compatible with MLFlow. We know, we know, everyone hates when developers invent new APIs, but you can rest easy because SynapseML integrates cleanly into existing Spark ML APIs so you can embed models directly into existing pipelines. We strive to make SynapseML available to developers wherever they work, and the library is available in a variety of languages like Python, Scala, Java, R. As of this release SynapseML is also usable from .NET, C#, F#.</P> <P>&nbsp;</P> <P><FONT size="4"><STRONG>Highlights in this release</STRONG></FONT></P> <P>&nbsp;</P> <P><STRONG>.NET, C#, and F# Support</STRONG></P> <P>Dear .NET community: we heard you! In SynapseML v0.10, we are adding full support for .NET languages like C# and F#. This means that you can now use everything in SynapseML from any of the .NET ecosystem languages, and even load up models you built in other languages like Python, R, and Java.</P> <P>&nbsp;</P> <P>For a quick taste of the new .NET bindings, you can train a distributed LightGBM model with no fuss:</P> <P>&nbsp;</P> <LI-CODE lang="applescript">var fitModel = new LightGBMClassifier() .SetObjective("binary") .SetNumLeaves(30) .SetNumIterations(200) .Fit(trainDf); </LI-CODE> <P>&nbsp;</P> <P>View our <A href="#" target="_blank" rel="noopener">.NET getting started page</A> for more details.</P> <P>&nbsp;</P> <P><STRONG>OpenAI Language Models</STRONG></P> <P>SynapseML also offers a simple and scalable way to leverage Azure Cognitive Services directly from Spark.</P> <P>Currently, SynapseML supports over 50 Cognitive Services and are now expanding that the <A href="#" target="_blank" rel="noopener">new Azure Open AI Service</A>. This service allows users to tap into 175-Billion parameter language models (GPT-3) from OpenAI that can generate and complete text and code near human parity. GPT-3 requires only a small amount of input text to generate large volumes of relevant and sophisticated machine-generated text. GPT-3 can also <A href="#" target="_blank" rel="noopener">solve a variety of ML tasks</A> beyond just completion including summarization, translation, sentiment analysis, analogical reasoning, question answering, and more.</P> <P>&nbsp;</P> <P>To learn more, check out <A href="#" target="_blank" rel="noopener">the SynapseML OpenAI guide</A> or check out a demonstration showing how simple it is to <A href="#" target="_blank" rel="noopener">create a search engine on custom unstructured data using SynapseML and GPT-3</A>.</P> <P>&nbsp;</P> <P><STRONG>Full Support for MLflow</STRONG></P> <P><A href="#" target="_blank" rel="noopener">MLflow</A> is a platform for managing the machine learning lifecycle and streamline machine learning development, including tracking experiments, packaging code into reproducible runs, and sharing and deploying models. We are very happy to announce that SynapseML models now integrate with MLflow with full support for saving, loading, deployment, and autologging. This means that you can now manage the lifecycle of your SynapseML models the same way that you manage your other models using MLflow. Finally, not only did we enable full support for SynapseML, but we contributed back to the MLFlow project to enable autologging for all SparkML models!</P> <P>For more information, check out the <A href="#" target="_blank" rel="noopener">MLflow in SynapseML getting started guide</A> and our <A href="#" target="_blank" rel="noopener">documentation on MLflow autologging</A>.<BR /><BR /></P> <P><STRONG>No Cluster, no Problem: Experiment in Browser with Binder</STRONG></P> <P>We know that Spark can be intimidating for first users but fear not because with the technology <A href="#" target="_blank" rel="noopener">Binder</A>, you can explore and experiment with SynapseML with zero setup, install, infrastructure, or Azure account required. Simply visit <A href="#" target="_blank" rel="noopener">our Binder site</A> to get started in your browser!</P> <P>&nbsp;</P> <P><STRONG>There is more…</STRONG></P> <P>Finally, in addition to the highlights mentioned above, there are many other great updates in this release for Responsible AI, Azure Cognitive Services, LightGBM on Spark, Wowpal Wabbit and other features. We could not capture all of it here but you can read more about this release in the detailed <A href="#" target="_blank" rel="noopener">release notes</A> and learn all about the rich SynapseML capabilities <A href="#" target="_blank" rel="noopener">on our website.</A></P> <P>&nbsp;</P> <P>We would also like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML. We encourage you to learn more about these amazing developers through <A href="#" target="_blank" rel="noopener">our contributor spotlight</A>.</P> <P>&nbsp;</P> <P>We really hope you enjoy the new version of SynapseML and don’t hesitate to reach out to let us know what you think! Stay tuned for more exciting updates and let us know you appreciate SynapseML by giving us a star on <A href="#" target="_blank" rel="noopener">GitHub</A>. ;)</img></P> <P>&nbsp;</P> <P><EM>- Mark Hamilton, Senior Software Engineer, Azure Synapse&nbsp;</EM></P> <P><EM>-Nellie Gustafsson, Principal Product Manager, Azure Synapse</EM></P> <P>&nbsp;</P> Tue, 09 Aug 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/exciting-new-release-of-synapseml/ba-p/3589606 Nellie Gustafsson 2022-08-09T15:00:00Z Connect DEP Enabled Synapse Spark to On-Prem Apache Kafka https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/connect-dep-enabled-synapse-spark-to-on-prem-apache-kafka/ba-p/3379023 <P><STRONG>Connecting On-Prem Apache Kafka from Azure Synapse Spark Notebook from DEP Enabled and Managed VNET-based Synapse Workspace</STRONG></P> <P><STRONG>&nbsp;</STRONG></P> <P><STRONG>Background</STRONG></P> <P>&nbsp;</P> <P><SPAN>Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines and streaming analytics with real-time to near real-time capabilities. There are many on-premises Apache Kafka implementations for various mission-critical applications.&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN>Azure Synapse Analytics is a limitless analytics service that brings together Enterprise Data Warehousing (EDW) and Big Data Analytics (BDA). In Azure Synapse Analytics, Microsoft provides its own implementation of Apache Spark. Azure Synapse Analytics also supports provisioning the workspace inside a Managed Virtual Network (Vnet) as well as enabling Data Exfiltration Protection to enhance the security. &nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN>However, if your Synapse Workspace is using a Managed Vnet and has Data Exfiltration Protection enabled you will not be able to connect to your On-Premises data sources. Also, there is no connector available in Azure Data Factory (ADF) or Synapse Pipeline for Apache Kafka. You can, however, integrate to On-Premises Apache Kafka using either Spark Structured Streaming API or any other Kafka Producer/Consumer supported libraries.</SPAN></P> <P>&nbsp;</P> <P><SPAN>Another options is that Azure Synapse Analytics also provides a Managed Private Endpoint, which is an option when you want to connect to the outside world from a Managed Vnet-based Synapse Workspace. Currently, Managed Private Endpoint does not have direct support for on-premises Apache Kafka either. Therefore, you will need to take a different route to connect an On-Prem Apache Kafka from Synapse Spark which is inside a Managed Vnet.</SPAN></P> <P>&nbsp;</P> <P><SPAN>The easiest and most commonly suggested approach is to use Apache Kafka Mirror Maker to mirror the On-Prem Apache Kafka topics to Azure Event Hubs. You can get more details from this link:&nbsp;</SPAN><SPAN><A href="#" target="_blank" rel="noopener">Use Apache Kafka MirrorMaker - Azure Event Hubs - Azure Event Hubs | Microsoft Docs</A>. While this works fine, it adds additional complexities to the data pipelines and overall data architecture. If we find a way to connect directly to an On-Prem Apache Kafka then it makes the data pipeline a little bit simpler and will also save time and money. This blog describes how to establish this connectivity and run the Synapse Spark Notebook to connect to On-Premises Apache Kafka directly. </SPAN></P> <P><STRONG>&nbsp;</STRONG><STRONG>&nbsp;</STRONG></P> <P><STRONG>Motivation</STRONG></P> <P>&nbsp;</P> <P><SPAN>Microsoft's official documentation for Azure Data Factory contains a tutorial which explains how to access an On-Premises SQL Server from Azure Data Factory which is inside a Managed Vnet. You can go through that article here:&nbsp;</SPAN><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed Vnet using Private Endpoint - Azure Data Factory | Microsoft Docs</A>.</SPAN></P> <P>&nbsp;</P> <P><SPAN>Although based upon the article's solution, to meet our requirements we needed to substitute On-Prem Apache Kafka for On-Prem SQL Server and instead of an Azure Data Factory inside a Managed Vnet, we used a Synapse Workspace inside a Managed Vnet. The "Forwarding Vnet" concept explained in the above tutorial remains as-is in our approach.</SPAN></P> <P>&nbsp;</P> <P><STRONG>Approach</STRONG></P> <P>&nbsp;</P> <P><SPAN>An Azure Synapse Analytics workspace supports creating a Managed Private Endpoint using Private Link Service. This will allow you to define Fully Qualified Domain Names (FQDNs) of your data sources to you connect to from Synapse Workspace. Our approach is to define FQDNs of our On-Prem Apache Kafka. The Private Link Service will then rely on a Load Balancer to direct the traffic based upon load balancing rules to the backend subnet consisting of two NAT VMs. These VMs will have On-Premises connectivity via Express Route or any other network mechanism. Using this architecture, our requests to access Apache Kafka hosted on On-Premises network will pass through smoothly.</SPAN>&nbsp;</P> <DIV id="tinyMceEditorBhadreshShiyal_0" class="mceNonEditable lia-copypaste-placeholder">&nbsp;</DIV> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="BhadreshShiyal_4-1657109029670.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386088i785CB6A64063E83C/image-size/large?v=v2&amp;px=999" role="button" title="BhadreshShiyal_4-1657109029670.png" alt="BhadreshShiyal_4-1657109029670.png" /></span></P> <P>&nbsp;</P> <P><STRONG>Setup &amp; Configuration </STRONG></P> <P>&nbsp;</P> <P><STRONG>A. Create Subnets </STRONG></P> <P><SPAN>You can follow the instructions from the below mentioned URL for this.</SPAN><SPAN>&nbsp;</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P>&nbsp;</P> <P><STRONG>B. Create a standard Internal Load Balancer</STRONG></P> <P><SPAN>You can follow the instructions from the below mentioned URL for this.</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P>&nbsp;</P> <P><STRONG>C. Create Load Balancer Resources</STRONG></P> <P><STRONG>1. Create a backend pool </STRONG></P> <P><SPAN>You can follow the instructions here:</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P><STRONG>2. Create a health probe</STRONG></P> <P><SPAN>You can follow the instructions here:</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P><STRONG>3. Create a load balancer rule &nbsp;</STRONG></P> <P>A load balancer rule is used to define how traffic is distributed to the VMs. You define the frontend IP configuration for the incoming traffic and the backend IP pool to receive the traffic. The source and destination port are defined in the rule.</P> <P><STRONG>Note:</STRONG></P> <P>Here, the Port and Backend Port both are configured as <STRONG>9092</STRONG>. You should use the port which your Kafka Brokers are configured to use for communication.</P> <P>&nbsp;</P> <P><STRONG>Next, create a load balancer rule:</STRONG></P> <OL> <LI>Select&nbsp;<STRONG>All services</STRONG>&nbsp;in the left-hand menu, select&nbsp;<STRONG>All resources</STRONG>, and then select&nbsp;<STRONG>myLoadBalancer</STRONG>&nbsp;from the resources list.</LI> <LI>Under&nbsp;<STRONG>Settings</STRONG>, select&nbsp;<STRONG>Load-balancing rules</STRONG>, then select&nbsp;<STRONG>Add</STRONG>.</LI> <LI>Use these values to configure the load-balancing rule:</LI> </OL> <P>&nbsp;</P> <TABLE width="452"> <THEAD> <TR> <TD width="159"> <P><STRONG>Setting</STRONG></P> </TD> <TD width="293"> <P><STRONG>Value</STRONG></P> </TD> </TR> </THEAD> <TBODY> <TR> <TD width="159"> <P>Name</P> </TD> <TD width="293"> <P>Enter&nbsp;<STRONG>myRule</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>IP Version</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>IPv4</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Frontend IP address</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>LoadBalancerFrontEnd</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Protocol</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>TCP</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Port</P> </TD> <TD width="293"> <P>Enter&nbsp;<STRONG>9092</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Backend port</P> </TD> <TD width="293"> <P>Enter&nbsp;<STRONG>9092</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Backend pool</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>myBackendPool</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Health probe</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>myHealthProbe</STRONG>.</P> </TD> </TR> <TR> <TD width="159"> <P>Idle timeout (minutes)</P> </TD> <TD width="293"> <P>Move the slider to&nbsp;<STRONG>15</STRONG>&nbsp;minutes.</P> </TD> </TR> <TR> <TD width="159"> <P>TCP reset</P> </TD> <TD width="293"> <P>Select&nbsp;<STRONG>Disabled</STRONG>.</P> </TD> </TR> </TBODY> </TABLE> <P class="lia-indent-padding-left-30px">4. Leave the rest of the defaults and then select&nbsp;<STRONG>OK</STRONG>.</P> <P class="lia-indent-padding-left-30px">5. Create a separate Load Balancer Rule for each Kafka Broker by using a different backend port for each Kafka Broker. You can use any port number as far as you don’t repeat it for your setup. Please note that later will route all traffic to the same port as of your On-Premises Kafka Broker. This is just an intermediate arrangement so that Synapse Spark does not have a clear line of sight to each of your On-Prem Kafka Brokers.</P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P><STRONG>D. Create a private link service </STRONG></P> <P><SPAN>You can follow the instructions here:</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P><SPAN>You will need as many private link services as the number of on-premises Kafka brokers you want to connect to from a Synapse Spark notebook. Please refer to the above diagram. &nbsp;</SPAN></P> <P>&nbsp;</P> <P><STRONG>E. Create backend servers (NAT VMs) </STRONG></P> <P><SPAN>You can follow the instructions here:</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs</A></SPAN></P> <P><STRONG>Note:</STRONG></P> <P>After provisioning NAT VMs, please ensure that they have the ports open which you are going to use for health probe as well as for Kafka. If those ports are not open, then please add appropriate Inbound NSG rules. Without that, NAT VMs will not be able to communicate properly using a Forwarding Vnet.</P> <P><SPAN>&nbsp;</SPAN></P> <P><STRONG>F. Create Forwarding Rule(s) to Endpoint(s)</STRONG></P> <OL> <LI>Login and copy script&nbsp;<SPAN><A href="#" target="_blank" rel="noopener">ip_fwd.sh</A></SPAN>&nbsp;to your backend server VMs.</LI> <LI>Run the script on with the following options:<BR /><STRONG style="font-family: inherit;">sudo ./ip_fwd.sh -i eth0 -f 9092 -a &lt;FQDN/IP&gt; -b 9092</STRONG></LI> </OL> <P class="lia-indent-padding-left-30px">&nbsp; &nbsp;Here, &lt;FQDN/IP&gt; is your target Kafka Broker IP.</P> <P class="lia-indent-padding-left-30px">3. Run below command and check the iptables in your backend server VMs. You can see one record in your iptables with your target IP.</P> <P class="lia-indent-padding-left-30px"><STRONG>sudo iptables -t nat -v -L PREROUTING -n --line-number</STRONG></P> <P><STRONG>&nbsp;</STRONG></P> <P><STRONG>Note:</STRONG></P> <P>If you have more than one Kafka Broker, you will need to define multiple load balancer rule as each broker will have to use a different port to route the traffic to NAT VMs. There will be multiple IP table records with different ports. Otherwise, there will be some issues. For example,</P> <TABLE style="width: 700px;" width="700"> <THEAD> <TR> <TD width="141">&nbsp;</TD> <TD width="482"> <P><STRONG>Command run in backend server VM</STRONG></P> </TD> </TR> </THEAD> <TBODY> <TR> <TD width="141"> <P><STRONG>Kafka Broker 1</STRONG></P> </TD> <TD width="482"> <P>sudo ./ip_fwd.sh -i eth0 -f 9092 -a &lt;Kafka Broker 1 FQDN/IP&gt; -b 9092</P> </TD> </TR> <TR> <TD width="141"> <P><STRONG>Kafka Broker 2</STRONG></P> </TD> <TD width="482"> <P>sudo ./ip_fwd.sh -i eth0 -f 9093 -a &lt;Kafka Broker 2 FQDN/IP&gt; -b 9092</P> </TD> </TR> <TR> <TD width="141"> <P><STRONG>Kafka Broker 3</STRONG></P> </TD> <TD width="482"> <P>sudo ./ip_fwd.sh -i eth0 -f 9094 -a &lt;Kafka Broker 3 FQDN/IP&gt; -b 9092</P> </TD> </TR> </TBODY> </TABLE> <P><STRONG><SPAN>&nbsp;</SPAN></STRONG></P> <P><STRONG><SPAN>G.&nbsp;</SPAN></STRONG><STRONG>Create a Managed Private Endpoint to Private Link Service</STRONG></P> <OL> <LI>Select All services in the left-hand menu, select All resources, and then select your data factory from the resources list.</LI> <LI>Select&nbsp;<STRONG>Author &amp; Monitor</STRONG>&nbsp;to launch the Data Factory UI in a separate tab.</LI> <LI>Go to the&nbsp;<STRONG>Manage</STRONG>&nbsp;tab and then go to the&nbsp;<STRONG>Managed private endpoints</STRONG>&nbsp;section.</LI> <LI>Select +&nbsp;<STRONG>New</STRONG>&nbsp;under&nbsp;<STRONG>Managed private endpoints</STRONG>.</LI> <LI>Select the&nbsp;<STRONG>Private Link Service</STRONG>&nbsp;tile from the list and select&nbsp;<STRONG>Continue</STRONG>.</LI> <LI>Enter the name of private endpoint and select&nbsp;<STRONG>myPrivateLinkService</STRONG>&nbsp;in private link service list.</LI> <LI>Add FQDN of your target on-premises Kafka Broker.</LI> </OL> <P><STRONG>Notes:</STRONG></P> <UL> <LI>It accepts only FQDNs so you will not be able to add IP Addresses of your On-Prem Kafka Cluster. If your On-Prem Kafka Cluster does not have FQDNs defined for each of the brokers then as an alternative you can make a host file entry with pseudo FQDNs for each of the On-Prem Kafka Brokes in both NAT VMs.</LI> </UL> <P>&nbsp; &nbsp; &nbsp; 8. Create private endpoint.</P> <P><STRONG><SPAN>Note:</SPAN></STRONG></P> <UL> <LI><SPAN>You <STRONG>CANNOT</STRONG> create a Linked Service and test the connection as a Kafka Connector is not available for Azure Data Factory or Synapse Pipelines. </SPAN></LI> <LI>If there are more than one Kafka Brokers in your On-Prem Kafka Cluster, then please ensure that you create one Private Links Service for each of your On-Prem Kafka Brokers.</LI> </UL> <P><STRONG><SPAN>&nbsp;H.&nbsp;</SPAN></STRONG><STRONG>Create a Synapse Spark Notebook</STRONG></P> <P><SPAN>Here, we assume you already have the Synapse Spark Pool created to attach and run the notebook. If that is not the case then please follow the link here to create the Synapse Spark Pool first:</SPAN><SPAN>&nbsp;<A href="#" target="_blank" rel="noopener">Quickstart: Create a serverless Apache Spark pool using the Azure portal - Azure Synapse Analytics | Microsoft Docs</A></SPAN></P> <P>&nbsp;</P> <P><SPAN>For using the Spark Structured Streaming API with Kafka, you need to have spark-sql-kafka-0-10_2.11-2.4.8.jar installed on your Synapse Spark Pool. As our Synapse Workspace is behind a managed Vnet without internet access, you will need to download the appropriate version of this jar from this maven repository link (</SPAN><SPAN><A href="#" target="_blank" rel="noopener">Maven Repository: org.apache.spark » spark-sql-kafka-0-10)&nbsp; (mvnrepository.com)</A> and upload it to the workspace packages.</SPAN></P> <P><SPAN>Once it is available in Synapse Studio’s workspace packages section, you can install it to the Synapse Spark Pool using the Synapse Studio GUI. Please refer below URL if you need additional information on package management in Synapse Workspace:</SPAN></P> <P><SPAN><A href="#" target="_blank" rel="noopener">Package management - Azure Synapse Analytics | Microsoft Docs</A></SPAN></P> <P><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </SPAN></P> <P><SPAN>Now, we are ready to create a Synapse Spark Notebook. Let us look at that. &nbsp;&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN>We will create Synapse Spark Notebook to test the connectivity. There are a couple of ways you can test the connectivity to On-Prem Kafka. We will try to use Spark’s Structured Streaming API first:</SPAN></P> <UL> <LI><SPAN>Create a new notebook and enter the following code in its&nbsp;</SPAN>cells. <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="BhadreshShiyal_5-1657109545591.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386091iB62F3BD899D0B8FA/image-size/large?v=v2&amp;px=999" role="button" title="BhadreshShiyal_5-1657109545591.png" alt="BhadreshShiyal_5-1657109545591.png" /></span></LI> <LI><SPAN>The above code snippet will create a streaming dataframe using Spark’s Structured Streaming API. You should replace values of kafka.bootstrap.servers and subscribe option as per your setup. &nbsp;&nbsp;</SPAN></LI> </UL> <P class="lia-indent-padding-left-30px"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="BhadreshShiyal_6-1657109612907.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386092iC1C4778E8EC38F38/image-size/large?v=v2&amp;px=999" role="button" title="BhadreshShiyal_6-1657109612907.png" alt="BhadreshShiyal_6-1657109612907.png" /></span></P> <UL> <LI><SPAN>The above code snippet will create a query which will write the streaming messages to the specified path on your Azure Data Lake Storage location along with updating the checkpoint location. Please ensure you change the values for the path and checkpointLocation as per your setup. The writeStream command is responsible for writing your streaming data to Azure Data Lake Storage on the given path in the format which you have specified in the&nbsp;</SPAN>command.</LI> </UL> <P class="lia-indent-padding-left-30px"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="BhadreshShiyal_7-1657109665276.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386093iCF17933470F6DF41/image-size/large?v=v2&amp;px=999" role="button" title="BhadreshShiyal_7-1657109665276.png" alt="BhadreshShiyal_7-1657109665276.png" /></span></P> <UL> <LI><SPAN>You can try running the above cells one by one to know the status of your streaming queries. If your query is active then it will show you true as an output for the query.isActive cell. If your query is processing some messages then it will show the details of the same as output for query.recentProgress and query.lastProgress</SPAN></LI> </UL> Mon, 01 Aug 2022 19:57:48 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/connect-dep-enabled-synapse-spark-to-on-prem-apache-kafka/ba-p/3379023 BhadreshShiyal 2022-08-01T19:57:48Z Azure Synapse Analytics July Update 2022 https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-july-update-2022/ba-p/3535089 <DIV id="generatedtoc"> <H2><SPAN data-contrast="none">Azure Synapse Analytics July Update 2022</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> </DIV> <P><SPAN data-contrast="auto">Welcome to the July 2022 update for Azure Synapse Analytics! This month, you will find information about Apache Spark 3.2 general availability, performance/cost improvements to Synapse Link for SQL, and support for Data Explorer as an output connection on Azure Stream Analytics jobs. Additional general updates and new features in Spark, Synapse Data Explorer, Data Integration, Developer Experience and Security are also mentioned in this edition.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Don’t forget to check out our companion video!</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/04EB6HBkWGY?mute=1&amp;autoplay=1" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <H1>Table of contents</H1> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_0" target="_self" rel="nofollow noopener noreferrer"> Apache Spark for Synapse&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_1" target="_self" rel="nofollow noopener noreferrer">Synapse runtime for Spark 3.2 is now generally available&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_2" target="_self" rel="nofollow noopener noreferrer"> Intelligent Cache is now generally available&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_3" target="_self" rel="nofollow noopener noreferrer"> Synapse runtime for Apache Spark 2.4 officially enters its retirement lifecycle&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_4" target="_self" rel="nofollow noopener noreferrer"> Synapse Data Explorer&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_5" target="_self" rel="nofollow noopener noreferrer"> Ingest data from Azure Stream Analytics into Synapse Data Explorer (Preview)&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_6" target="_self" rel="nofollow noopener noreferrer"> Synapse Web Data Explorer now supports rendering charts for each y column&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_7" target="_self" rel="nofollow noopener noreferrer"> Create tables graphically in Synapse Web Data Explorer&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_8" target="_self" rel="nofollow noopener noreferrer"> Data Integration&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_9" target="_self" rel="nofollow noopener noreferrer"> Time-To-Live in managed VNET [Public Preview]&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_10" target="_self" rel="nofollow noopener noreferrer"> SAP Change Data Capture (CDC) connector is now available in [Public Preview]&nbsp;&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_11" target="_self" rel="nofollow noopener noreferrer"> Synapse Link&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_12" target="_self" rel="nofollow noopener noreferrer"> Reduce cost with batch mode in Synapse link for SQL&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_13" target="_self" rel="nofollow noopener noreferrer"> Developer Experience&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_14" target="_self" rel="nofollow noopener noreferrer"> Synapse Notebooks is now fully compatible with IPython&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_15" target="_self" rel="nofollow noopener noreferrer"> Mssparkutils now has spark.stop() method&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_16" target="_self" rel="nofollow noopener noreferrer"> Security&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_17" target="_self" rel="nofollow noopener noreferrer"> Changes to permissions needed for publishing to Git&nbsp;</A></LI> </UL> </LI> </UL> <P>&nbsp;</P> <H2 id="TOCREF_0" aria-level="2"><STRONG><SPAN data-contrast="none">Apache Spark for Synapse</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_1" aria-level="2"><SPAN data-contrast="none">Synapse runtime for Spark 3.2 is now generally available</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Apache Spark 3.2 for Synapse has been in public preview for two months. After considerable customer feedback and notable improvements in performance and stability, Azure Synapse runtime for Apache Spark 3.2 now becomes generally available and ready for production workloads.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Here’s what got better:&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI><SPAN data-contrast="auto">Delta Lake version 1.2 brings new features like bin-compaction, data skipping using statistics and generated columns, lightweight column renaming, and many more. For further details, please read the full </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Delta Lake 1.2 release notes</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">The pandas API on Apache Spark helps unify small data and big data APIs.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Adaptive Query Execution to speed up Spark SQL at runtime, along with query compilation and shuffle performance improvements.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Extended ANSI SQL compatibility mode to simplify migration of SQL workloads.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Here is the list of specific changes:&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI><SPAN data-contrast="auto">Support Pandas API layer on PySpark (SPARK-34849)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Enable adaptive query execution by default (SPARK-33679)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Query compilation latency reduction (SPARK-35042, SPARK-35103, SPARK-34989)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Support push-based shuffle to improve shuffle efficiency (SPARK-30602)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Add RocksDB StateStore implementation (SPARK-34198)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">EventTime based session windows (SPARK-10816)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">ANSI SQL mode GA (SPARK-35030)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI><SPAN data-contrast="auto">Support for ANSI SQL INTERVAL types (SPARK-27790)</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P>&nbsp;</P> <P><SPAN data-contrast="auto">For detailed contents and lifecycle information on all runtimes, you can visit </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Apache Spark runtimes in Azure Synapse</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_2" aria-level="2"><SPAN data-contrast="none">Intelligent Cache is now generally available</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">The Intelligent Cache works seamlessly behind the scenes and caches data to speed up Spark execution as it reads from your ADLS Gen2 data lake. The feature helps reduce your total cost of ownership by improving performance up to 65% on subsequent reads of cached files for Parquet files and 50% for CSV files. This feature is now available for all Synapse runtimes for Apache Spark version 3 (3.1 and 3.2), and you can enable it in the Synapse Spark pool configuration.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Intelligent Cache in Azure Synapse Analytics</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_3" aria-level="2"><SPAN data-contrast="none">Synapse runtime for Apache Spark 2.4 officially enters its retirement lifecycle</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Back in May 2021, we announced the retirement of Apache Spark 2.4 in favor of Apache Spark 3.2. We extended Apache Spark 2.4 based on customer feedback. With the general availability of the Apache Spark 3.2 runtime, we are officially announcing a retirement date for the Azure Synapse runtime for Apache Spark 2.4.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Starting July 2022, a full 12-month retirement cycle begins. You should relocate your workloads to the newer Apache Spark 3.2 runtime within this period. To understand the Apache Spark lifecycle and support policies as well as impact on workloads, you can read more at </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Apache Spark runtimes in Azure Synapse</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H2 id="TOCREF_4" aria-level="2"><STRONG><SPAN data-contrast="none">Synapse Data Explorer</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P><SPAN data-contrast="auto">Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. It offers ingestion from Event Hubs, IoT Hubs, blobs written to blob containers, and Azure Stream Analytics jobs.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_5" aria-level="3"><SPAN data-contrast="none">Ingest data from Azure Stream Analytics into Synapse Data Explorer (Preview)</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Stream Analytics</SPAN></A> <SPAN data-contrast="auto">is a real-time analytics service and an event-processing engine designed to analyze and process high volumes of fast streaming data from multiple sources simultaneously. An Azure Stream Analytics job consists of an input source, a transformation query, and an output connection. There are several output types to which you can send transformed data.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_10-1658953297461.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391274i8175DC8038B2BA34/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_10-1658953297461.png" alt="thynguyen_10-1658953297461.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">You can now use a Streaming Analytics job to collect data from an event hub and send it to your Azure Data Explorer cluster using the Azure portal or an ARM template</SPAN><SPAN data-contrast="none">. </SPAN><SPAN data-contrast="none">For full details, please read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Ingest data from Azure Stream Analytics into Azure Data Explorer</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_6"><SPAN data-contrast="none">Synapse Web Data Explorer now supports rendering charts for each y column</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Starting this month, we have added support for the y-split property when using the render operator for bar, column, time, line, and area charts in Synapse Web Explorer. When implementing ysplit=panels, one chart is rendered for each ycolumn value (up to some limit), as you can see in the example below:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="sql">let min_t = datetime(2017-01-05); let max_t = datetime(2017-02-03 22:00); let dt = 2h; demo_make_series2 | make-series num=avg(num) on TimeStamp from min_t to max_t step dt by sid | where sid == 'TS1' // select a single time series just to get cleaner visualization | extend (baseline, seasonal, trend, residual) = series_decompose(num, -1, 'linefit') | render timechart with( ysplit=panels) </LI-CODE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_11-1658953297469.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391275i0B9BF33FB66C718F/image-size/large?v=v2&amp;px=999" role="button" title="thynguyen_11-1658953297469.png" alt="thynguyen_11-1658953297469.png" /></span></P> <H3 id="TOCREF_7"><SPAN data-contrast="none">Create tables graphically in Synapse Web Data Explorer</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Previously, you could create a table in Synapse Web Data Explorer using ingestion by inferring its structure from sample data provided in a blob, ADLS, or Event Hubs. Starting this month, you can create tables graphically without providing data.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The new source type “None” triggers the flow.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_12-1658953297472.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391276iC83013109D39CBC4/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_12-1658953297472.png" alt="thynguyen_12-1658953297472.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">On the schema tab, you can add columns and select data types to create a new table graphically.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_13-1658953297473.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391282iA0BB9717D1548745/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_13-1658953297473.png" alt="thynguyen_13-1658953297473.png" /></span></P> <P>&nbsp;</P> <H3 id="TOCREF_8" aria-level="2"><STRONG><SPAN data-contrast="none">Data Integration</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <H3 id="TOCREF_9"><SPAN data-contrast="none">Time-To-Live in managed VNET [Public Preview]</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">We have introduced the public preview of time-to-live (TTL) in managed virtual network. With this, you can reserve computes for the TTL period, saving time and improving efficiency.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_14-1658953297475.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391280i291279099FAE9DC8/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_14-1658953297475.png" alt="thynguyen_14-1658953297475.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-factory-blog/announcing-public-preview-of-time-to-live-ttl-in-managed-virtual/ba-p/3552879" target="_blank" rel="noopener"><SPAN data-contrast="none">Announcing public preview of Time-To-Live (TTL) in managed virtual network</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <H3 id="TOCREF_10"><SPAN data-contrast="none">SAP Change Data Capture (CDC) connector is now available in [Public Preview]&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">We have released a new data connector which can connect to all SAP systems that support ODP and can be used with predefined templates to help streamline the integration of SAP data within Azure services. You can now extract data changes instead of using workarounds like watermarking to extract new or updated records.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_15-1658953297476.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391281i68BF2984C0E82B7D/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_15-1658953297476.png" alt="thynguyen_15-1658953297476.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more, see </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-factory-blog/announcing-public-preview-of-the-sap-cdc-solution-in-azure-data/ba-p/3420904" target="_blank" rel="noopener"><SPAN data-contrast="none">Announcing the Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:257}">&nbsp;</SPAN></P> <H2 id="TOCREF_11" aria-level="2"><STRONG><SPAN data-contrast="none">Synapse Link</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_12" aria-level="3"><SPAN data-contrast="none">Reduce cost with batch mode in Synapse link for SQL</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Azure Synapse Link for SQL is an automated system for replicating data from your transactional databases into a dedicated SQL pool in Azure Synapse Analytics. Starting this month, you can make trade-offs between cost and latency in Synapse Link for SQL by selecting the continuous or batch mode to replicate your data.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">By selecting “continuous mode”, the runtime will be running continuously so that any changes applied to the SQL database or SQL Server will be replicated to Synapse with low latency. Alternatively, when you select “batch mode” with a specified interval, the changes applied to the SQL database or SQL Server will be accumulated and replicated to Synapse in batch mode with the specified interval. This can save cost as you are only charged for the time the runtime is required to replicate data. After each batch of data is replicated, the runtime will be shut down automatically.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more about Synapse link for SQL, please read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">What is Azure Synapse Link for SQL?</SPAN></A><SPAN data-contrast="auto"> and </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/synapse-link-for-sql-deep-dive/ba-p/3567645" target="_blank" rel="noopener"><SPAN data-contrast="none">Synapse Link for SQL Deep Dive</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H2 id="TOCREF_13"><STRONG><SPAN data-contrast="none">Developer Experience</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_14" aria-level="2"><SPAN data-contrast="none">Synapse Notebooks is now fully compatible with IPython</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">The official kernel for Jupyter notebooks is IPython and it is now supported in Synapse Notebooks. Previously, Synapse Notebooks did not support IPython. When running PySpark notebooks, IPython will be used as the default Python interpreter.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Here’s what got better for you:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI><SPAN data-contrast="auto">Dozens of new </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">built-in</SPAN></A><SPAN data-contrast="auto"> magic commands.&nbsp;Register your custom magics and use them&nbsp;in any language context.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_16-1658953297479.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391287iD33CF20522D14FD1/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_16-1658953297479.png" alt="thynguyen_16-1658953297479.png" /></span></P> <P>&nbsp;</P> <UL> <LI><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN><SPAN data-contrast="auto">Better debug experience with in-line error highlighting.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-contrast="auto">Before:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_17-1658953297481.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391286i5D2D7C95B21772D8/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_17-1658953297481.png" alt="thynguyen_17-1658953297481.png" /></span></P> <P><SPAN data-contrast="auto">After:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_18-1658953297484.jpeg" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391288i10C4615428C98CD1/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_18-1658953297484.jpeg" alt="thynguyen_18-1658953297484.jpeg" /></span></P> <P>&nbsp;</P> <UL> <LI><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN><SPAN data-contrast="auto">Richer auto-completion experience.</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="thynguyen_19-1658953297486.jpeg" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/391290i9E08FEF38113ED5D/image-size/medium?v=v2&amp;px=400" role="button" title="thynguyen_19-1658953297486.jpeg" alt="thynguyen_19-1658953297486.jpeg" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_15"><SPAN data-contrast="none">Mssparkutils now has spark.stop() method</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">A new API </SPAN><I><SPAN data-contrast="auto">mssparkutils.session.stop()</SPAN></I><SPAN data-contrast="auto"> has been added to the mssparkutils package. With this new API, you can programmatically stop your interactive session. You can release resources as soon as their notebook is finished running. This feature becomes handy when there are multiple sessions running against the same Spark pool. The new API is available for Scala and Python.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more please read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Stop an interactive session</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H2 id="TOCREF_16" aria-level="2"><STRONG><SPAN data-contrast="none">Security</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_17" aria-level="2"><SPAN data-contrast="none">Changes to permissions needed for publishing to Git</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Previously, to commit changes to the current branch in Synapse, you needed Git permissions and the Azure Contributor role (Azure RBAC) in addition to the Synapse Artifact Publisher role (Synapse RBAC). With this enhancement, you will now need only Git permissions and the Synapse Artifact Publisher (Synapse RBAC) role to commit changes in Git-mode. To learn more, please read</SPAN> <A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Access control enforcement in Synapse Studio</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:270}">&nbsp;</SPAN></P> <P>&nbsp;</P> Thu, 28 Jul 2022 17:12:02 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-july-update-2022/ba-p/3535089 ryanmajidi 2022-07-28T17:12:02Z How to use CETAS on serverless SQL pool to improve performance and automatically recreate it https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/how-to-use-cetas-on-serverless-sql-pool-to-improve-performance/ba-p/3548040 <P>Serverless SQL pool &nbsp;has a very cool way to handle CREATE EXTERNAL TABLE AS SELECT (CETAS).&nbsp; I mean, you can use a serverless SQL pool to create a CETAS which will materialize the query results. It means a heavy query in which, for example, the results would be part of future large join with other queries or aggregations that can be consolidated for reporting purposes. Those scenarios are examples that could be used as a CETAS. CETAS will be stored as a parquet file accessed by an external table in your storage and the performance is awesome. Later I am suggesting how to recreate the CETAS using pipeline against serverless SQL Pool.</P> <P>&nbsp;</P> <P><STRONG>CETAS on serverless SQL Pool</STRONG></P> <P>&nbsp;</P> <P>Let's consider the following&nbsp;query scenario as an example, every object part of this query could represent a View which is been executed against the Storage. There are some <STRONG>aggregations</STRONG> in this query and those results will be used in a <STRONG>report</STRONG> to filter later, also that data does not need to be updated frequently. Just once a day.</P> <P>&nbsp;</P> <P><STRONG>Query:</STRONG></P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript"> SELECT Dsr.SalesReasonName , COUNT_BIG(distinct Fis.SalesOrderNumber) SalesOrderNumber_COUNT , AVG(CAST(SalesAmount AS DECIMAL(38,4))) SalesAmount_AVG , AVG(CAST(OrderQuantity AS DECIMAL(38,4))) OrderQuantity_AVG FROM ViewFactSale AS FIS INNER JOIN ViewFactSaleReason AS Fisr ON Fisr.SalesOrderNumber = Fis.SalesOrderNumber AND Fisr.SalesOrderLineNumber = Fis.SalesOrderLineNumber INNER JOIN ViewDimSales AS Dsr ON Fisr.SalesReasonKey = Dsr.SalesReasonKey GROUP BY Fis.SalesTerritoryKey, Fis.OrderDateKey, Dsr.SalesReasonName</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>The view I am using is querying some delta files. However, it could be a parquet, CSV o JSON.</P> <P>Information about the format supported datatypes:&nbsp;<A href="#" target="_blank" rel="noopener">Serverless SQL pool - Azure Synapse Analytics | Microsoft Docs</A></P> <P>&nbsp;</P> <P><STRONG>Do you need the View to use CETAS? No</STRONG>. You could use Select Openrowset directly inside of the CETAS or the View. However, note I will be using scoped credentials and authenticating my query with MSI because <STRONG>I am impersonating the permissions to access the storage account through the MSI</STRONG> and that will be important later.</P> <P>&nbsp;</P> <P>More information? Check on the Doc.:</P> <P><A href="#" target="_blank" rel="noopener">CREATE EXTERNAL TABLE AS SELECT (CETAS) in Synapse SQL - Azure Synapse Analytics | Microsoft Docs</A></P> <P><A href="#" target="_blank" rel="noopener">Create and use views in serverless SQL pool - Azure Synapse Analytics | Microsoft Docs</A></P> <P>&nbsp;</P> <P>Anyway, back to our T-SQL query example, here is the step by step to create the script:</P> <P>&nbsp;</P> <P>1) First, I defined the database scoped credential. I choose MSI which means managed identity.</P> <P>2) Second, I created the external data source pointing to the storage files under the scoped I just defined.</P> <P>3) Third,&nbsp; the view definition using the data source created.</P> <P>&nbsp;</P> <P><STRONG>T-SQL example:</STRONG></P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">///Scoped credential to authenticate against storage CREATE DATABASE SCOPED CREDENTIAL MSI_Conn WITH IDENTITY = 'Managed Identity' ///Storage CREATE EXTERNAL DATA SOURCE Storage WITH ( LOCATION = 'https://Storageaccountname.dfs.core.windows.net/Container/Folder/', CREDENTIAL = [MSI_Conn] //View CREATE VIEW ViewDimSales AS SELECT * FROM OPENROWSET( BULK '/DimSales', FORMAT = 'Delta', DATA_SOURCE = 'Storage' ) AS DimSales </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>As the definition of the database scoped is MSI as I just mentioned, it means my Synapse Workspace needs to have Storage Blob Data contributor permission on the Storage Account (Fig 1) to be queried,&nbsp; hence when I execute this query it will use MSI workspace permissions.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_0-1656084133013.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383382i7654B6AF3590487B/image-size/medium?v=v2&amp;px=400" role="button" title="Liliam_Leme_0-1656084133013.png" alt="Liliam_Leme_0-1656084133013.png" /></span></P> <P>Fig 1</P> <P>&nbsp;</P> <P>Some references for more information related to the storage permissions and scoped credential</P> <P><A href="#" target="_blank" rel="noopener">CREATE DATABASE SCOPED CREDENTIAL (Transact-SQL) - SQL Server | Microsoft Docs</A></P> <P><A href="#" target="_blank" rel="noopener">Control storage account access for serverless SQL pool - Azure Synapse Analytics | Microsoft Docs</A></P> <P>&nbsp;</P> <P><STRONG>OK, now let me create my CETAS.</STRONG>&nbsp;So before creating the CETAS I need to define a format file and a data source to store my CETAS files and here goes the steps:</P> <P>&nbsp;</P> <OL> <LI><SPAN>Create the database scoped credential ( reusing the one&nbsp; from the last example above)</SPAN></LI> <LI><SPAN>Create the external file format ( I choose parquet for performance reasons)</SPAN></LI> <LI><SPAN>Create the external Datasource</SPAN></LI> <LI><SPAN>Create the external table (CETAS)</SPAN></LI> </OL> <P>&nbsp;</P> <LI-CODE lang="applescript"> //Format for my CETAS CREATE EXTERNAL FILE FORMAT Parquet_file WITH ( FORMAT_TYPE = PARQUET ) //Storage path where the result set will be materialized CREATE EXTERNAL DATA SOURCE Storage WITH ( LOCATION = 'https://StorageAccountName.dfs.core.windows.net/Container/Folder/CETAS', CREDENTIAL = [MSI_Conn] ) //CETAS CREATE EXTERNAL TABLE FactSale_CETAS WITH ( LOCATION = 'FactSale_CETAS/', DATA_SOURCE = Storage, FILE_FORMAT = Parquet_file ) AS SELECT Dsr.SalesReasonName , COUNT_BIG(distinct Fis.SalesOrderNumber) SalesOrderNumber_COUNT , AVG(CAST(SalesAmount AS DECIMAL(38,4))) SalesAmount_AVG , AVG(CAST(OrderQuantity AS DECIMAL(38,4))) OrderQuantity_AVG FROM ViewFactSale AS FIS INNER JOIN ViewFactSaleReason AS Fisr ON Fisr.SalesOrderNumber = Fis.SalesOrderNumber AND Fisr.SalesOrderLineNumber = Fis.SalesOrderLineNumber INNER JOIN ViewDimSales AS Dsr ON Fisr.SalesReasonKey = Dsr.SalesReasonKey GROUP BY Fis.SalesTerritoryKey, Fis.OrderDateKey, Dsr.SalesReasonName </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>CETAS can drastically improve the performance of your query</STRONG>. For example, this view execution in my environment took 34 seconds to complete,&nbsp; while when I change this execution to materialize the aggregations with CETAS it took 4 seconds.</P> <P>&nbsp;</P> <P>CETAS&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Views only</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_1-1657096050974.png" style="width: 526px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386047i7223D2F11378B61D/image-dimensions/526x357?v=v2" width="526" height="357" role="button" title="Liliam_Leme_1-1657096050974.png" alt="Liliam_Leme_1-1657096050974.png" /></span></P> <P>&nbsp;</P> <P>However, if there is a need to update regularly this information. Would be just dropping and recreating?</P> <P>Answer: yes… almost.</P> <P>&nbsp;</P> <P><STRONG>Recreating CETAS</STRONG></P> <P>&nbsp;</P> <P><SPAN>DROP EXTERNAL TABLE is a metadata operation, so once the external table is dropped&nbsp;</SPAN>on the serverless side the file would still exist on the Storage Account.</P> <P>&nbsp;</P> <P>As for the example, it will be:</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">DROP EXTERNAL TABLE [FactSale_CETAS]</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Therefore, you will need to go there and delete the folder and files to recreate it again (manually), otherwise, you will hit an error like this:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_0-1656081740782.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383379iF03315EA1608CE57/image-size/medium?v=v2&amp;px=400" role="button" title="Liliam_Leme_0-1656081740782.png" alt="Liliam_Leme_0-1656081740782.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Fig 2</P> <P>&nbsp;</P> <P><STRONG>Pipelines&nbsp; as Answer:&nbsp;</STRONG></P> <P>&nbsp;</P> <P>So here is an idea of how to customize this process with Pipelines… and it is pretty easy (Fig 3).</P> <P>&nbsp;</P> <P>1) Create an Activity to Delete Files</P> <P>2) Create a Stored Procedure with the CETAS re-creation definition on <SPAN>serverless SQL pool</SPAN></P> <P>3) Call this Stored Procedure from the Pipeline</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_0-1656079744771.png" style="width: 650px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383366i53DB773E770AD0C8/image-dimensions/650x195?v=v2" width="650" height="195" role="button" title="Liliam_Leme_0-1656079744771.png" alt="Liliam_Leme_0-1656079744771.png" /></span></P> <P>Fig 3</P> <P>&nbsp;</P> <P><STRONG>Step by step:&nbsp;</STRONG></P> <P>1)<STRONG> Delete Files</STRONG>:&nbsp; Add the <STRONG>Activity</STRONG> called <STRONG>Delete</STRONG> and configure it pointing to the folder where the CETAS was configured to be stored. This will delete the files and folder, so be careful to not point to the wrong folder( Fig 4 and 5):.</P> <P>&nbsp;</P> <P>In my example, the path would be as follows, so I configured it to the folder&nbsp;FactSale_CETAS:</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">https://StorageAccountName.dfs.core.windows.net/Container/Folder/CETAS/FactSale_CETAS</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_4-1656080052923.png" style="width: 497px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383371i5BBF02B26451F969/image-dimensions/497x399?v=v2" width="497" height="399" role="button" title="Liliam_Leme_4-1656080052923.png" alt="Liliam_Leme_4-1656080052923.png" /></span></P> <P>Fig 4</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_3-1656080005764.png" style="width: 563px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383370i53A89A2E7D9B458C/image-dimensions/563x221?v=v2" width="563" height="221" role="button" title="Liliam_Leme_3-1656080005764.png" alt="Liliam_Leme_3-1656080005764.png" /></span></P> <P>Fig 5</P> <P>&nbsp;</P> <P>2) <STRONG>Stored Procedure:&nbsp;</STRONG><STRONG>Add the CETAS</STRONG> re-creation inside of a <STRONG>Stored Procedure</STRONG> inside of a Serverless database.&nbsp;</P> <P>Follow my script example:</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">CREATE PROCEDURE fact_ctas_renew AS BEGIN IF EXISTS ( SELECT 1 FROM SYS.TABLES WHERE NAME = 'FactSale_CETAS') BEGIN DROP EXTERNAL TABLE FactSale_CETAS END ELSE BEGIN CREATE EXTERNAL TABLE FactSale_CETAS WITH ( LOCATION = 'FactSale_CETAS/', DATA_SOURCE = Storage, FILE_FORMAT = Parquet_file ) AS SELECT Dsr.SalesReasonName , COUNT_BIG(distinct Fis.SalesOrderNumber) SalesOrderNumber_COUNT , AVG(CAST(SalesAmount AS DECIMAL(38,4))) SalesAmount_AVG , AVG(CAST(OrderQuantity AS DECIMAL(38,4))) OrderQuantity_AVG FROM ViewFactSale AS FIS INNER JOIN ViewFactSaleReason AS Fisr ON Fisr.SalesOrderNumber = Fis.SalesOrderNumber AND Fisr.SalesOrderLineNumber = Fis.SalesOrderLineNumber INNER JOIN ViewDimSales AS Dsr ON Fisr.SalesReasonKey = Dsr.SalesReasonKey GROUP BY Fis.SalesTerritoryKey, Fis.OrderDateKey, Dsr.SalesReasonName END END </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>3) <STRONG>Execute the Stored Procedure</STRONG>: If <STRONG>you need to execute an activity against a&nbsp;<SPAN>serverless SQL pool</SPAN>, choose SQL Stored Procedure under General Activity</STRONG> ( Fig 6 and 7). While doing that I created a linked service pointing to SQL Server as the example below. Note,&nbsp; even though my User and Password are SQL (Fig 7), my <STRONG>View definition is using the Data Source with MSI scope</STRONG>( look at view definition above in case of doubts), so the permissions to connect to the storage while executing the queries are MSI not SQL. Even because it is not possible to add a SQL User in an Azure Storage Account.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_1-1656079829742.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383368i40721C18FE3215C8/image-size/medium?v=v2&amp;px=400" role="button" title="Liliam_Leme_1-1656079829742.png" alt="Liliam_Leme_1-1656079829742.png" /></span></P> <P>Fig 6</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_2-1656079897751.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/383369iDD651C3AE2FEF8F0/image-size/medium?v=v2&amp;px=400" role="button" title="Liliam_Leme_2-1656079897751.png" alt="Liliam_Leme_2-1656079897751.png" /></span></P> <P><SPAN>&nbsp; &nbsp;Fig 7</SPAN></P> <P>&nbsp;</P> <P><SPAN> Now you can run CETAS over and over again!</SPAN></P> <P>&nbsp;</P> <P><U><STRONG>Conclusion</STRONG></U>:</P> <P>&nbsp;</P> <P>CETAS is an easy way to consolidate information for an intermediate query step ( when a sub-query result will be used to join other tables in a complex query join) or an even query with multiple aggregations. It improves the performance&nbsp;<SPAN>and cost, as subsequent queries will process fewer data.</SPAN>&nbsp;I worked in scenarios where we changed a 30 min of execution timing out to seconds and the pipeline is one way out to make this process automatic and simple.</P> <P>&nbsp;</P> <P>I hope it helps,</P> <P>&nbsp;</P> <P>Liliam</P> <P>UK Engineer</P> Tue, 19 Jul 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/how-to-use-cetas-on-serverless-sql-pool-to-improve-performance/ba-p/3548040 Liliam_Leme 2022-07-19T15:00:00Z Synapse Link for SQL Deep Dive https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/synapse-link-for-sql-deep-dive/ba-p/3567645 <P>Azure Synapse Link for SQL greatly simplifies analytics pipelines as Microsoft manages the orchestration process for you. Since being announced at Microsoft Build, many of you have had the opportunity to try it out in a POC so now seems like a good time to take a deep dive on some implementation aspects that may save you time later.</P> <P>&nbsp;</P> <P>This blog post will assume you have experience with Synapse dedicated SQL pools and that you have some basic working knowledge of Synapse Link for SQL from following the <A href="#" target="_blank" rel="noopener">quick start</A> or from doing an initial POC.</P> <P>&nbsp;</P> <H1>Known Limitations</H1> <P>Synapse Link for SQL is currently in Preview, and as with all preview services, there are limitations you should be aware of prior to use. Review the <A href="#" target="_blank" rel="noopener">known limitations</A> page before starting any projects. The product will continue to improve over time as it moves to general availability.</P> <P>&nbsp;</P> <H1>Ingestion Service New Features</H1> <P>Speaking of that constant improvement journey, you can now select from a wider variety of core counts when establishing the link. Most notably the option for 4 cores (2 + 2 driver cores), which would cut the minimum ingestion cost (previously 8 cores) in half.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_0-1657206797202.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386387iEB3BD22384981CE3/image-size/medium?v=v2&amp;px=400" role="button" title="stevehow_0-1657206797202.png" alt="stevehow_0-1657206797202.png" /></span></P> <P>&nbsp;</P> <P>There is also a new feature to select a batch mode for ingestion. Batch mode allows you to save even more on costs by only paying for ingestion service during the batch loads instead of it being continuously on. You can select between 20 and 60 minutes for batch processing.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_1-1657206797206.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386388i332815DC0B88FAAB/image-size/medium?v=v2&amp;px=400" role="button" title="stevehow_1-1657206797206.png" alt="stevehow_1-1657206797206.png" /></span></P> <P>&nbsp;</P> <H1>Concurrency</H1> <P>In Synapse link for SQL, the ingestion service is by default “aggressive” in providing the lowest latency possible for syncing data. However, there are concurrency implications.</P> <P>&nbsp;</P> <P>At different service level objectives (SLOs), Synapse dedicated SQL pools have an increasing number of concurrency slots available. As found in this <A href="#" target="_blank" rel="noopener">chart</A>, a 100 DWU dedicated SQL pool has 4 concurrency slots while a DWU 30000 has 128. In pools with lower SLOs, setting up Synapse Link for SQL could increase that number of queued queries when using the defaults.</P> <P>&nbsp;</P> <P>Below is a link configuration for three identical tables representing NYC taxi trip data. This link is running on a minimum 100 DWU dedicated SQL pool which gives 4 total concurrency slots.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_2-1657206797210.png" style="width: 832px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386389i5DA7467CA950B4F0/image-dimensions/832x198?v=v2" width="832" height="198" role="button" title="stevehow_2-1657206797210.png" alt="stevehow_2-1657206797210.png" /></span></P> <P>&nbsp;</P> <P>In this scenario, I am sending hundreds of taxi trip events per minute (using event hub and stream analytics) to all three Azure SQL DB tables simultaneously.</P> <P>&nbsp;</P> <P>When I start the link, these tables will be processed concurrently. Three tables would create three concurrent connections. We can validate that behavior below.</P> <P>&nbsp;</P> <P>In the Azure Portal, there is a metric for workload group allocation. Synapse Link for SQL uses the default “smallrc” workload group (see <A href="#" target="_blank" rel="noopener">concurrency limits</A> doc). At 100 DWU, there are 4 concurrency slots available so in the below chart, the maximum concurrency being achieved is 75% of the total, or 3 of 4 slots.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_3-1657206797215.png" style="width: 876px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386390iE846951EC64129F3/image-dimensions/876x322?v=v2" width="876" height="322" role="button" title="stevehow_3-1657206797215.png" alt="stevehow_3-1657206797215.png" /></span></P> <P>&nbsp;</P> <P>There is fluctuation here based on the ingestion service processing of each table, however 50-75% or 2-3 slots respectively is constantly being allocated to Synapse Link for SQL.&nbsp;If we normally have 3 or 4 active user queries running at a time on this dedicated SQL pool, then they would all be competing for the 1-2 slots not being used by Synapse Link for SQL.</P> <P>&nbsp;</P> <P><STRONG>So what do we do?</STRONG></P> <P>Rather than increasing the DWU, there is a setting in Synapse Link for SQL that enables transaction consistency across tables.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_4-1657206797217.png" style="width: 610px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386391i4FBF31A5F4153C4E/image-dimensions/610x302?v=v2" width="610" height="302" role="button" title="stevehow_4-1657206797217.png" alt="stevehow_4-1657206797217.png" /></span></P> <P>&nbsp;</P> <P>Enabling transaction consistency also has the biproduct of changing the processing logic from concurrent to sequential. Latency will increase between SQL DB and Synapse, but now only one concurrency slot is being used.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_5-1657206797226.png" style="width: 874px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386392iF32891202F7CA4E6/image-dimensions/874x518?v=v2" width="874" height="518" role="button" title="stevehow_5-1657206797226.png" alt="stevehow_5-1657206797226.png" /></span></P> <P>&nbsp;</P> <P>The above chart shows only 25% allocation (1 of 4 slots) for smallrc.</P> <P>&nbsp;</P> <P>For smaller dedicated SQL pools, running Synapse Link for SQL with “transaction consistency” is a good practice and will allow processing of dozens of tables without consuming dozens of slots.</P> <P>&nbsp;</P> <H1>Snapshot Isolation and Dirty Reads</H1> <P>To see how Synapse Link for SQL is applying inserts, updates, and deletes to the dedicated sql pool, you can simply look in sys.dm_pdw_exec_requests or view the recent sql requests in the Monitor Hub.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_6-1657206797233.png" style="width: 785px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386393i81054E50A2A37A97/image-dimensions/785x595?v=v2" width="785" height="595" role="button" title="stevehow_6-1657206797233.png" alt="stevehow_6-1657206797233.png" /></span></P> <P>&nbsp;</P> <P>QID10618 from the screenshot above is below SQL.</P> <P>&nbsp;</P> <LI-CODE lang="sql">COPY INTO [dbo].[#source_23d9f25bf6104b31ac82c1ac42cd1cfa_insert] FROM 'https://&lt;blobstorage&gt;.blob.core.windows.net/********/indexWorkingDir/0000029f00000ec00016-16c447dc2bca4be8bebc24fe0777f3a6/insert.index' WITH (FILE_TYPE = 'CSV', FIELDQUOTE = '0X22', FIELDTERMINATOR=',', FILE_SPLIT = 'OFF', CREDENTIAL = (IDENTITY='Shared Access Signature', SECRET='***') );</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>You can see that a staging table is being populated from a file in blob storage called “insert.index”. If there were updates or deletes being processed there would be presumably another file called “delete.index” as well.</P> <P>&nbsp;</P> <P>The workflow here is that all updates are replaced with delete/insert pairs and then all deletes and inserts are applied to separate temp tables using the copy command. Then a join delete and bulk insert from select take place to update the target tables defined in the link.</P> <P>These are all implementation details that could change at any time, but they can plainly be identified from reviewing the previously run queries.</P> <P>&nbsp;</P> <P>This all takes place in one transaction. However, some customers have noticed when running updates through the link, such as the scenario below, that they can see the dirty reads within the transaction boundaries.</P> <P>&nbsp;</P> <LI-CODE lang="sql">-- run on source database update nyctaxitrips set trip_distance = trip_distance + 0.1, [EventProcessedUtcTime] = GETUTCDATE() where dropoff_location_id = 232</LI-CODE> <P>&nbsp;</P> <P>(1737 rows affected)</P> <P>&nbsp;</P> <LI-CODE lang="sql">-- run on target database select count(*) as ROW_COUNT from NYCTaxiTrips where dropoff_location_id = 232</LI-CODE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_7-1657206797233.png" style="width: 240px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386394i66305C4B37EF2AC4/image-dimensions/240x104?v=v2" width="240" height="104" role="button" title="stevehow_7-1657206797233.png" alt="stevehow_7-1657206797233.png" /></span></P> <P>&nbsp;</P> <LI-CODE lang="sql">-- run on target database select count(*) as ROW_COUNT from NYCTaxiTrips where dropoff_location_id = 232</LI-CODE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_8-1657206797234.png" style="width: 244px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386395iF73021FC6118EB34/image-dimensions/244x129?v=v2" width="244" height="129" role="button" title="stevehow_8-1657206797234.png" alt="stevehow_8-1657206797234.png" /></span></P> <P>&nbsp;</P> <P>This is simply because many POC environments for Synapse dedicated SQL pools have default settings, one of which is <A href="#" target="_blank" rel="noopener">read_committed_snapshot</A> being set to OFF.</P> <P>&nbsp;</P> <LI-CODE lang="sql">--SNAPSHOT ISOLATION LEVEL select [name], is_read_committed_snapshot_on from sys.databases ALTER DATABASE [sqlpool] SET READ_COMMITTED_SNAPSHOT ON;</LI-CODE> <P>&nbsp;</P> <P>Ensure that snapshot isolation is ON, then rerun your tests and the dirty reads are no longer allowed.</P> <P>&nbsp;</P> <H1>Stopping and Restarting your Link</H1> <P>One great feature of Synapse Link for SQL is that new tables can be added, or existing tables can be deleted, without having to stop your link.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_9-1657206797237.png" style="width: 513px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386396i13D2790D3E35E499/image-dimensions/513x299?v=v2" width="513" height="299" role="button" title="stevehow_9-1657206797237.png" alt="stevehow_9-1657206797237.png" /></span></P> <P>&nbsp;</P> <P>However, if you want to change the properties of the ingestion service, such as batch mode, or increasing or decreasing the core count, you must stop the link.</P> <P>&nbsp;</P> <P>After your changes are made, publish them, and restart the link. It is not obvious from the link dialog that there may be errors, use the “monitor this link connection” shortcut at the top of the dialog or go to the Monitor Hub Link Connections pane.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_10-1657206797237.png" style="width: 221px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386397i8958763860831E22/image-dimensions/221x52?v=v2" width="221" height="52" role="button" title="stevehow_10-1657206797237.png" alt="stevehow_10-1657206797237.png" /></span></P> <P>&nbsp;</P> <P>Unfortunately, the existing tables in the link have errors.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_11-1657206797240.png" style="width: 659px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386398i8B631AF11A710C97/image-dimensions/659x140?v=v2" width="659" height="140" role="button" title="stevehow_11-1657206797240.png" alt="stevehow_11-1657206797240.png" /></span></P> <P>&nbsp;</P> <P>There is a checkbox “Drop and recreate table on target” on the link connection that can fix this.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_12-1657206797242.png" style="width: 591px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386399i4DA4928F2308F562/image-dimensions/591x325?v=v2" width="591" height="325" role="button" title="stevehow_12-1657206797242.png" alt="stevehow_12-1657206797242.png" /></span></P> <P>&nbsp;</P> <P>When this box is checked, Synapse Link for SQL will drop and recreate the target table, then get a snapshot of all records in the source tables and bulk load them. When restarting with this checkbox checked, everything will work as expected.</P> <P>&nbsp;</P> <P>As shown earlier, we can look at the query executions during snapshot and see the path is now “FullCopyData/data-n.parquet”.</P> <P>&nbsp;</P> <LI-CODE lang="sql">COPY INTO [dbo].[source_23d9f25bf6104b31ac82c1ac42cd1cfa_ef17514b-4a2e-4a90-8831-5915342e022d_snapshot_ef5075ce48034c36a1e9b14feca4efb5] FROM 'https://&lt;blobstorage&gt;.blob.core.windows.net/*****/Tables/fde74837-ddbf-4755-b700-3db93a5a594f/TableData_000001af000016280002/FullCopyData/data-1.parquet' WITH (FILE_TYPE = 'PARQUET', FILE_SPLIT = 'OFF', CREDENTIAL = (IDENTITY='Shared Access Signature', SECRET='***') , AUTO_CREATE_TABLE = 'ON'); COPY INTO [dbo].[source_23d9f25bf6104b31ac82c1ac42cd1cfa_ef17514b-4a2e-4a90-8831-5915342e022d_snapshot_76cf8ee0ecc04d67b35489772e539b4c] FROM 'https://&lt;blobstorage&gt;.blob.core.windows.net/*****/Tables/fde74837-ddbf-4755-b700-3db93a5a594f/TableData_000001af000016280002/FullCopyData/data-2.parquet' WITH (FILE_TYPE = 'PARQUET', FILE_SPLIT = 'OFF', CREDENTIAL = (IDENTITY='Shared Access Signature', SECRET='***') , AUTO_CREATE_TABLE = 'ON');</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>What are the implications of this?</STRONG></P> <P>Most obviously, large tables must be reloaded which could take much longer than smaller tables to complete.</P> <P>As Synapse can support multiple links to the same source database, one idea would be for very large tables, establish a dedicated link.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="stevehow_13-1657206797244.png" style="width: 657px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/386400i29524A1ADA0FE8E3/image-dimensions/657x133?v=v2" width="657" height="133" role="button" title="stevehow_13-1657206797244.png" alt="stevehow_13-1657206797244.png" /></span></P> <P>Now when updates have to be made to the main link with multiple tables, it won’t disrupt the “largetablelink” that I have exclusively setup for dbo.NYCTaxiTrips4.</P> <P>&nbsp;</P> <P>With the addition of lower core counts, and batch mode in the ingestion service, having multiple links becomes more feasible from a cost perspective.</P> <P>&nbsp;</P> <H1>Conclusion</H1> <P>Synapse Link for SQL will continue to evolve as moving from Preview to GA. Continue to watch for updates in this space.</P> <P>&nbsp;</P> Tue, 12 Jul 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/synapse-link-for-sql-deep-dive/ba-p/3567645 stevehow 2022-07-12T15:00:00Z Azure Synapse Analytics - How to activate my Synapse workspace using ARM templates ? https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-how-to-activate-my-synapse-workspace/ba-p/3565748 <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A>&nbsp;is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, manage and serve data for immediate BI and machine learning needs.</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Azure Resource Manager</A> is the deployment and management service for Azure and the ARM templates allow you to create and deploy an entire Azure infrastructure declaratively. For example, you can deploy not only virtual machines, but also the network infrastructure, storage systems, and any other resources you may need.</P> <P><BR />ARM template is a JavaScript Object Notation (JSON) file that defines one or more resources to deploy to a resource group, subscription, management group, or tenant. The template can be used to deploy the resources consistently and repeatedly. (<A href="#" target="_blank" rel="noopener">more</A>)</P> <P>When the Synapse workspace is created with the encryption that is referred to Azure KeyVault key, the workspace needs to be “Activated” before moving forward with other deployments such as Dedicated SQL pools, Apache Spark Pools.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_0-1657032863747.png" style="width: 676px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/385816i7FADEFC3A053CD06/image-dimensions/676x80?v=v2" width="676" height="80" role="button" title="CharithCaldera_0-1657032863747.png" alt="CharithCaldera_0-1657032863747.png" /></span></P> <P>&nbsp;</P> <P>There are multiple ways that the workspace can be activated, but there are multiple steps that needs to be completed before activating the workspace, (<A href="#" target="_blank" rel="noopener">more</A>).</P> <P>&nbsp;</P> <P>Example:&nbsp;<BR /><BR /></P> <OL> <LI>Confirm the Azure KeyVault is created with “purge protection”.<BR /><A href="#" target="_blank" rel="noopener">Purge protection</A> is an optional Key Vault behavior and is not enabled by default. Purge protection can only be enabled once soft delete is enabled. It can be turned on via CLI or PowerShell.</LI> <LI>Grant the Workspace managed Identity required permissions (Get, Wrap Key, Unwrap Key) on Azure KeyVault.</LI> <LI>Manually activate the workspace.</LI> </OL> <P>&nbsp;</P> <P>Even If the KeyVault is already deployed, there are multiple use cases related to the automations and activating the Synapse Workspaces such as,<BR /><BR /></P> <OL> <LI>Create access policies to Synapse workspace managed identity during the deployments.</LI> <LI>Provide proper permissions (Get, Wrap Key, Unwrap Key) to workspace managed identity during the deployments.</LI> <LI>Activate the Synapse workspace without any manual action.</LI> <LI>Complete all above actions (#1, #2, #3) in a single run.</LI> </OL> <P>There are multiple cmdlets and commands provided in Azure PowerShell and Azure CLI to achieve this and activate the workspace successfully, but that requires to run multiple commands time to time.</P> <P>&nbsp;</P> <P>Example:</P> <P>In Azure PowerShell,</P> <P>“<A href="#" target="_blank" rel="noopener">Enable-AzSynapseWorkspace</A>” activates the Synapse Workspace, but the access policies need to be configured before running this command.<BR /><BR /></P> <P>In Azure CLI,</P> <P>“<A href="#" target="_blank" rel="noopener">az synapse workspace activate</A>” activates the Synapse Workspace, but the access policies need to be configured before running this command.</P> <P>&nbsp;</P> <P><STRONG><I>The Azure Resource Manager Templates (ARM templates)</I></STRONG><EM><STRONG>&nbsp;can be used to perform all of these actions as one single task easily and activate the workspace during the deployment process.<BR /><BR /></STRONG></EM>The template file contains the deployment operations, and the Parameter file contains the values that required for the deployments. <BR />The template file also can contain the different variables that can be an output of the previous operation. (Example: Managed Identity Object).<EM><STRONG><BR /></STRONG></EM></P> <P>&nbsp;</P> <P>In this example, we assume that the Azure KeyVault is already deployed under the same resource group and below Template and Parameter files can be used to deploy the Synapse workspace &amp; activate at the same time.</P> <P>&nbsp;</P> <P>Example: <BR />(Following example template file can be used to deploy a Synapse workspace and activate using a key in AzureKeyVault)</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="json">{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.1", "parameters": { "name": { "type": "String" }, "location": { "type": "String" }, "defaultDataLakeStorageAccountName": { "type": "String" }, "defaultDataLakeStorageFilesystemName": { "type": "String" }, "sqlAdministratorLogin": { "type": "String" }, "sqlAdministratorLoginPassword": { "defaultValue": "", "type": "SecureString" }, "setWorkspaceIdentityRbacOnStorageAccount": { "type": "Bool" }, "createManagedPrivateEndpoint": { "type": "Bool" }, "azureADOnlyAuthentication": { "type": "Bool" }, "allowAllConnections": { "defaultValue": true, "type": "Bool" }, "managedVirtualNetwork": { "allowedValues": [ "default", "" ], "type": "String" }, "tagValues": { "defaultValue": {}, "type": "Object" }, "useSystemAssignedIdentity": { "type": "Bool" }, "storageSubscriptionID": { "defaultValue": "[subscription().subscriptionId]", "type": "String" }, "storageLocation": { "defaultValue": "[resourceGroup().location]", "type": "String" }, "storageRoleUniqueId": { "defaultValue": "[newGuid()]", "type": "String" }, "isNewStorageAccount": { "defaultValue": false, "type": "Bool" }, "isNewFileSystemOnly": { "defaultValue": false, "type": "Bool" }, "storageAccessTier": { "type": "String" }, "storageAccountType": { "type": "String" }, "storageSupportsHttpsTrafficOnly": { "type": "Bool" }, "storageKind": { "type": "String" }, "minimumTlsVersion": { "type": "String" }, "storageIsHnsEnabled": { "type": "Bool" }, "userObjectId": { "defaultValue": "", "type": "String" }, "setSbdcRbacOnStorageAccount": { "defaultValue": false, "type": "Bool" }, "setWorkspaceMsiByPassOnStorageAccount": { "defaultValue": false, "type": "Bool" }, "workspaceStorageAccountProperties": { "defaultValue": {}, "type": "Object" }, "keysPermissions": { "type": "array" }, "keyVaultName": { "type": "string" }, "keyVaultKeyName": { "type": "string" }, "workspaceKeyName": { "type": "string" } }, "variables": { "storageBlobDataContributorRoleID": "ba92f5b4-2d11-453d-a403-e96b0029c9fe", "defaultDataLakeStorageAccountUrl": "[concat('https://', parameters('defaultDataLakeStorageAccountName'), '.dfs.core.windows.net')]", "keyVaultURI": "[concat('https://', parameters('keyVaultName'), '.vault.azure.net', '/keys/', parameters('keyVaultKeyName'))]", "defaultAdlsGen2AccountResourceId": "[concat('/subscriptions/', parameters('storageSubscriptionID'), '/resourceGroups/',resourceGroup().name,'/providers/Microsoft.Storage/storageAccounts/',parameters('defaultDataLakeStorageAccountName'))]", "managedResourceGroupName": "[concat(resourceGroup().name, '-mrg')]" }, "resources": [ { "type": "Microsoft.Synapse/workspaces", "apiVersion": "2021-06-01", "name": "[parameters('name')]", "location": "[parameters('location')]", "dependsOn": [ "[concat('Microsoft.Storage/storageAccounts/', parameters('defaultDataLakeStorageAccountName'))]", "[concat('Microsoft.Resources/deployments/', parameters('defaultDataLakeStorageFilesystemName'))]" ], "tags": "[parameters('tagValues')]", "identity": { "type": "SystemAssigned" }, "properties": { "defaultDataLakeStorage": { "accountUrl": "[variables('defaultDataLakeStorageAccountUrl')]", "filesystem": "[parameters('defaultDataLakeStorageFilesystemName')]", "resourceId": "[variables('defaultAdlsGen2AccountResourceId')]", "createManagedPrivateEndpoint": "[parameters('createManagedPrivateEndpoint')]" }, "managedVirtualNetwork": "[parameters('managedVirtualNetwork')]", "managedResourceGroupName": "[variables('managedResourceGroupName')]", "azureADOnlyAuthentication": "[parameters('azureADOnlyAuthentication')]", "sqlAdministratorLogin": "[parameters('sqlAdministratorLogin')]", "sqlAdministratorLoginPassword": "[parameters('sqlAdministratorLoginPassword')]", "encryption": { "cmk": { "key": { "name": "[parameters('workspaceKeyName')]", "keyVaultUrl": "[variables('keyVaultURI')]" }, "keyIdentity": { "useSystemAssignedIdentity": "[parameters('useSystemAssignedIdentity')]" } } } }, "resources": [ { "type": "firewallrules", "apiVersion": "2021-06-01", "name": "allowAll", "location": "[parameters('location')]", "dependsOn": [ "[concat('Microsoft.Synapse/workspaces/', parameters('name'))]" ], "properties": { "startIpAddress": "0.0.0.0", "endIpAddress": "255.255.255.255" }, "condition": "[parameters('allowAllConnections')]" } ] }, { "type": "Microsoft.Resources/deployments", "apiVersion": "2021-04-01", "name": "StorageRoleDeploymentResource", "dependsOn": [ "[concat('Microsoft.Synapse/workspaces/', parameters('name'))]" ], "properties": { "mode": "Incremental", "template": { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {}, "variables": {}, "resources": [ { "type": "Microsoft.Storage/storageAccounts/providers/roleAssignments", "apiVersion": "2022-01-01-preview", "name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/Microsoft.Authorization/', guid(concat(resourceGroup().id, '/', variables('storageBlobDataContributorRoleID'), '/', parameters('name'), '/', parameters('storageRoleUniqueId'))))]", "location": "[parameters('storageLocation')]", "properties": { "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataContributorRoleID'))]", "principalId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2021-06-01', 'Full').identity.principalId]", "principalType": "ServicePrincipal" } }, { "condition": "[parameters('setSbdcRbacOnStorageAccount')]", "type": "Microsoft.Storage/storageAccounts/providers/roleAssignments", "apiVersion": "2022-01-01-preview", "name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/Microsoft.Authorization/', guid(concat(resourceGroup().id, '/', variables('storageBlobDataContributorRoleID'), '/', parameters('userObjectId'), '/', parameters('storageRoleUniqueId'))))]", "properties": { "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataContributorRoleID'))]", "principalId": "[parameters('userObjectId')]", "principalType": "User" } } ] } }, "subscriptionId": "[parameters('storageSubscriptionID')]", "resourceGroup": "[resourceGroup().name]", "condition": "[parameters('setWorkspaceIdentityRbacOnStorageAccount')]" }, { "type": "Microsoft.Resources/deployments", "apiVersion": "2021-04-01", "name": "UpdateStorageAccountNetworkingAcls", "dependsOn": [ "[concat('Microsoft.Synapse/workspaces/', parameters('name'))]" ], "properties": { "mode": "Incremental", "template": { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {}, "variables": {}, "resources": [ { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2021-01-01", "location": "[parameters('storageLocation')]", "name": "[parameters('defaultDataLakeStorageAccountName')]", "properties": "[parameters('workspaceStorageAccountProperties')]" } ] } }, "subscriptionId": "[parameters('storageSubscriptionID')]", "resourceGroup": "[resourceGroup().name]", "condition": "[parameters('setWorkspaceMsiByPassOnStorageAccount')]" }, { "type": "Microsoft.Storage/storageAccounts", "apiVersion": "2021-01-01", "name": "[parameters('defaultDataLakeStorageAccountName')]", "location": "[parameters('storageLocation')]", "tags": {}, "sku": { "name": "[parameters('storageAccountType')]" }, "kind": "[parameters('storageKind')]", "properties": { "accessTier": "[parameters('storageAccessTier')]", "supportsHttpsTrafficOnly": "[parameters('storageSupportsHttpsTrafficOnly')]", "isHnsEnabled": "[parameters('storageIsHnsEnabled')]", "minimumTlsVersion": "[parameters('minimumTlsVersion')]" }, "resources": [ { "type": "blobServices/containers", "apiVersion": "2021-01-01", "name": "[concat('default/', parameters('defaultDataLakeStorageFilesystemName'))]", "dependsOn": [ "[concat('Microsoft.Storage/storageAccounts/', parameters('defaultDataLakeStorageAccountName'))]" ], "properties": { "publicAccess": "None" }, "condition": "[parameters('isNewStorageAccount')]" } ], "condition": "[parameters('isNewStorageAccount')]" }, { "type": "Microsoft.Resources/deployments", "apiVersion": "2021-04-01", "name": "[parameters('defaultDataLakeStorageFilesystemName')]", "properties": { "mode": "Incremental", "template": { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {}, "variables": {}, "resources": [ { "type": "Microsoft.Storage/storageAccounts/blobServices/containers", "name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/default/', parameters('defaultDataLakeStorageFilesystemName'))]", "apiVersion": "2021-01-01", "properties": { "publicAccess": "None" } } ] } }, "subscriptionId": "[parameters('storageSubscriptionID')]", "resourceGroup": "[resourceGroup().name]", "condition": "[parameters('isNewFileSystemOnly')]" }, { "type": "Microsoft.KeyVault/vaults/accessPolicies", "name": "[concat(parameters('keyVaultName'), '/add')]", "apiVersion": "2021-10-01", "properties": { "accessPolicies": [ { "tenantId": "[subscription().tenantId]", "objectId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2019-06-01-preview', 'Full').identity.principalId]", "permissions": { "keys": "[parameters('keysPermissions')]" } } ] } }, { "type": "Microsoft.Synapse/workspaces/keys", "apiVersion": "2021-06-01", "name": "[concat(parameters('name'), '/',parameters('workspaceKeyName'))]", "dependsOn": [ "[resourceId('Microsoft.KeyVault/vaults/accessPolicies', parameters('keyVaultName'), 'add')]", "[concat('Microsoft.Synapse/workspaces/', parameters('name'))]" ], "properties": { "isActiveCMK": true, "keyVaultUrl": "[variables('keyVaultURI')]" } } ], "outputs": {} }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>There are multiple operations to be performed in this template such as creating storage, creating firewall rules and workspace, but since the access policies needs to be created before the activation, that is very important action to activate the workspace, the access policies are creating using below code snip.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="json"> { "type": "Microsoft.KeyVault/vaults/accessPolicies", "name": "[concat(parameters('keyVaultName'), '/add')]", "apiVersion": "2021-10-01", "properties": { "accessPolicies": [ { "tenantId": "[subscription().tenantId]", "objectId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2019-06-01-preview', 'Full').identity.principalId]", "permissions": { "keys": "[parameters('keysPermissions')]" } } ] } }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>The activation is mainly dependent on the policies created in Azure KeyVault. <BR />There are 2 properties on below – (isActiveCMK / keyVaultUrl) that needs to have proper parameters to activate the workspace.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="json"> { "type": "Microsoft.Synapse/workspaces/keys", "apiVersion": "2021-06-01", "name": "[concat(parameters('name'), '/',parameters('workspaceKeyName'))]", "dependsOn": [ "[resourceId('Microsoft.KeyVault/vaults/accessPolicies', parameters('keyVaultName'), 'add')]", "[concat('Microsoft.Synapse/workspaces/', parameters('name'))]" ], "properties": { "isActiveCMK": true, "keyVaultUrl": "[variables('keyVaultURI')]" } }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>During the deployment, the template file does not contain any hard coded values, the values are referred from the parameter file, or the values can be the outputs of deployments in ARM.</P> <P>&nbsp;</P> <P>Example: <BR />(Following example parameter file can be used to deploy a Synapse workspace and activate using a key in AzureKeyVault)</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="json">{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#", "contentVersion": "1.0.0.0", "parameters": { "name": { "value": "[Your Workspace Name]" }, "location": { "value": "[Region]" }, "defaultDataLakeStorageAccountName": { "value": "[Storage Account Name]" }, "defaultDataLakeStorageFilesystemName": { "value": "[Stroage Account File System]" }, "sqlAdministratorLogin": { "value": "[SQL Administrator Login]" }, "sqlAdministratorLoginPassword": { "value": "[SQL Administrator Password]" }, "keyVaultName": { "value": "[Azure KeyVault Name]" }, "setWorkspaceIdentityRbacOnStorageAccount": { "value": true }, "createManagedPrivateEndpoint": { "value": false }, "azureADOnlyAuthentication": { "value": false }, "allowAllConnections": { "value": true }, "managedVirtualNetwork": { "value": "" }, "tagValues": { "value": {} }, "keyVaultKeyName": { "value":"[Azure KeyVault Key name]" }, "workspaceKeyName": { "value":"[Azure KeyVault Key name]" }, "useSystemAssignedIdentity": { "value": true }, "storageSubscriptionID": { "value": "[Subscription ID]" }, "storageLocation": { "value": "[Storage Location]" }, "storageRoleUniqueId": { "value": "13c7d920-0450-4a62-baf8-37f4a367101c" }, "isNewStorageAccount": { "value": true }, "isNewFileSystemOnly": { "value": false }, "storageAccessTier": { "value": "Hot" }, "storageAccountType": { "value": "Standard_RAGRS" }, "storageSupportsHttpsTrafficOnly": { "value": true }, "storageKind": { "value": "StorageV2" }, "minimumTlsVersion": { "value": "TLS1_2" }, "storageIsHnsEnabled": { "value": true }, "userObjectId": { "value": "[User Object ID]" }, "setSbdcRbacOnStorageAccount": { "value": true }, "setWorkspaceMsiByPassOnStorageAccount": { "value": false }, "workspaceStorageAccountProperties": { "value": {} }, "keysPermissions": { "value": [ "get", "list", "wrapKey", "unwrapKey" ] } } }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>The parameter file contains multiple parameters that is used during the deployment such as,<BR /><BR /></P> <OL> <LI>Workspace name</LI> <LI>Location</LI> <LI>SQL admin login</LI> <LI>SQL password</LI> <LI>KeyVault name</LI> </OL> <P>As same as there are many deployment parameters mentioned. These parameters needs to be changed / modified based on your requirement.</P> <P>You can initiate the deployments using the parameter file and the template file from Azure PowerShell command “<A href="#" target="_blank" rel="noopener">New-AzResourceGroupDeployment</A>”</P> <P>&nbsp;</P> <P>Example: <BR />(Following example is to execute a ARM deployment using “New-AzResouceGroupDeployment” PowerShell cmdlet)</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="powershell">New-AzResourceGroupDeployment -Name "[Name for the deployment]" -TemplateFile [Template File Location]' -TemplateParameterFile '[Parameter File Location]' -ResourceGroupName "[ResourceGroup Name]"</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Monitor the Deployments:</STRONG></P> <P>&nbsp;</P> <P>The PowerShell cmdlet provides the output whether the deployment is success or not, but you can always monitor the deployment from the Azure portal by navigating in to the resource group.<BR />The ongoing deployments and the history of the deployments can be checked using the “Deployments".</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_0-1657034463854.png" style="width: 653px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/385824i63F69AB9F9676AA8/image-dimensions/653x142?v=v2" width="653" height="142" role="button" title="CharithCaldera_0-1657034463854.png" alt="CharithCaldera_0-1657034463854.png" /></span></P> <P>&nbsp;</P> <P>You can select the relevant deployment name from the list.&nbsp;</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_1-1657034538993.png" style="width: 652px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/385825i75732F02FAFBAD55/image-dimensions/652x103?v=v2" width="652" height="103" role="button" title="CharithCaldera_1-1657034538993.png" alt="CharithCaldera_1-1657034538993.png" /></span></P> <P>Deployment status:&nbsp;<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_2-1657034635300.png" style="width: 567px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/385826iE3793BCAA9525277/image-dimensions/567x236?v=v2" width="567" height="236" role="button" title="CharithCaldera_2-1657034635300.png" alt="CharithCaldera_2-1657034635300.png" /></span></P> <P>&nbsp;</P> <P>Let's&nbsp;recapitulate what we discussed above,&nbsp;<BR /><BR /></P> <P>When the Synapse workspace is created with the encryption that is referred to Azure KeyVault key, the workspace needs to be “Activated” before moving forward with other deployments such as Dedicated SQL pools, Apache Spark Pools.<BR />The administrators required to perform this action during the deployments as they use different techniques for automations.&nbsp;<BR />Azure PowerShell module and Azure CLI for Synapse has different set of commands to perform this action and also this activation can be done from Azure Portal manually.&nbsp;</P> <P>Not only from above, but this action can also be performed using ARM templates thought a single deployment without multiple actions.&nbsp;</P> Tue, 12 Jul 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-how-to-activate-my-synapse-workspace/ba-p/3565748 CharithCaldera 2022-07-12T15:00:00Z Ways you can browse industry templates and add tables to create your own lake database https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/ways-you-can-browse-industry-templates-and-add-tables-to-create/ba-p/3495011 <P>With Azure Synapse Database Templates generally available, our customers are constantly wanting to see and learn more about how to use these templates. Through these blogs we want to share tips and tricks our customers can use to help them utilize these templates in an efficient way. We’ve recently received several questions around the different ways a user can navigate these templates to create their lake databases. In this blog, I’d like to walk through a few options that may come handy as you give database templates a try.</P> <P>&nbsp;</P> <P>Azure Synapse Analytics offers a no-code database designer which allows you to browse these database templates, select and customize the tables you want to use, to model your enterprise data. There are several ways to browse the tables provided by the comprehensive industry templates within the designer’s exploration experience. Though the user experience is super intuitive, there are a few tips and tricks that can make this process even easier. Let’s do a quick tour to learn about the different ways to browse these templates.</P> <OL> <LI>Search for specific table names: This is the most basic option where a user can search using the table name as a keyword to bring back relevant results and select the appropriate table to add to the designer canvas.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_0-1656631037202.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384977iEA4D4B908FA7AD14/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_0-1656631037202.png" alt="lakshmimurthy_0-1656631037202.png" /></span></P> <P>&nbsp;</P> <OL start="2"> <LI>Using relationship toggle: This option comes handy when you have already selected the specific tables of interest and now want to bring in other tables that are related to a given table onto the canvas. One can easily click into the ‘Relationships’ tab of a table and toggle the ‘Show all’ button to view all the relationships to and from that table in a list view and select the appropriate relationships to add the corresponding table to the canvas.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_1-1656631058434.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384978iC45CF7DB7D9B98FF/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_1-1656631058434.png" alt="lakshmimurthy_1-1656631058434.png" /></span></P> <P>&nbsp;</P> <OL start="3"> <LI>Browse by business area and select the appropriate tables by checking the boxes: Users can expand a business area such as ‘Channel’ or ‘Business Metrics’ to view all the tables listed under that business area and select specific tables of interest to add to the canvas.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_2-1656631082745.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384979i771261083693939A/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_2-1656631082745.png" alt="lakshmimurthy_2-1656631082745.png" /></span></P> <P>&nbsp;</P> <OL start="4"> <LI>Select all tables within a business area: In this option, users can select &amp; add all tables shown under a business area to the canvas by clicking on the ellipsis next to the business area name and clicking on the ‘Show all’ option. If you ‘Unselect all’ then all the tables are removed from the canvas.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_3-1656631107266.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384980i19EEE6F6CBF12F72/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_3-1656631107266.png" alt="lakshmimurthy_3-1656631107266.png" /></span></P> <P>&nbsp;</P> <OL start="5"> <LI>Selecting an enterprise template: Clicking on the ‘Filter tables’ button lists the Enterprise templates available. Enterprise templates contain a subset of tables that describe the connectivity between the related business areas and are most likely to be of interest to an organization within a specific industry. They are a great starting point and serve as an accelerator for many types of large projects. To see all the tables related to a business area refer to 6) below.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_4-1656631129676.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384981i8DAB801F3D3E66E7/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_4-1656631129676.png" alt="lakshmimurthy_4-1656631129676.png" /></span></P> <P>&nbsp;</P> <OL start="6"> <LI>Selecting a business area template: Clicking on the ‘Filter tables’ button lists the Business Area templates available. Business Area templates describe a particular business or subject area with the greatest level of detail and lowest level of data granularity. They expand upon the subset of business area data represented in the industry enterprise database providing complete coverage for that business area. The tables in the Business Area template are a superset of the tables in the Business Area folder as they also contain related tables from other business areas to provide a complete the picture of the subject area.</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_5-1656631159545.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384982iFCFD2FC1E58C70E8/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_5-1656631159545.png" alt="lakshmimurthy_5-1656631159545.png" /></span></P> <P>&nbsp;</P> <OL start="7"> <LI>Add all related tables to a table on canvas: Another easy way to bring in all related tables to a table of interest is to click on ellipsis on the right corner of a table and click ‘Add related tables’ to bring in all the tables that have a relationship to &amp; from that table. A word of caution with this option is that in some cases this action might add over 100 tables to your canvas making it difficult to navigate. You can always use the appropriate zoom to traverse the tables. &nbsp;</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_6-1656631180543.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384983i7B999DC10BF2A5D3/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_6-1656631180543.png" alt="lakshmimurthy_6-1656631180543.png" /></span></P> <P>Did you know?</P> <P>In each of these scenarios, if you wish to undo one recent action you took, you can click on the ‘undo last action’ icon<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_0-1654896182557.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/379471iAFD80D6B392EA57D/image-size/medium?v=v2&amp;px=400" role="button" title="lakshmimurthy_0-1654896182557.png" alt="lakshmimurthy_0-1654896182557.png" /></span>. This is particularly useful when you ‘Add related tables’ and find that you don’t intend to keep all those tables on the canvas.</P> <P>&nbsp;</P> <P>To clear out your canvas and start over, you can click on the ellipsis icon&nbsp;<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lakshmimurthy_1-1654896214511.png" style="width: 47px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/379472i0DC8E551D070A149/image-dimensions/47x45?v=v2" width="47" height="45" role="button" title="lakshmimurthy_1-1654896214511.png" alt="lakshmimurthy_1-1654896214511.png" /></span>to start afresh.</P> <P>&nbsp;</P> <P>For more information about database templates or to get started, please visit <A href="#" target="_blank" rel="noopener">aka.ms/SynapseDBTemplates</A>.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> Wed, 06 Jul 2022 22:26:18 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/ways-you-can-browse-industry-templates-and-add-tables-to-create/ba-p/3495011 lakshmi-murthy 2022-07-06T22:26:18Z How to provide statement id in the serverless SQL pool support ticket? https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/how-to-provide-statement-id-in-the-serverless-sql-pool-support/ba-p/3532926 <P>Serverless SQL pool is a query service that enables you to analyze data in Azure Data Lake storage, Cosmos DB, or Data verse without a need to manage the resources, tune your queries, or do any other administration task to enable your data analyst to query data.</P> <P>&nbsp;</P> <P>There is a set of <A href="#" target="_blank" rel="noopener">best practices</A> that you should apply in your solution that should ensure optimal performance. By following the <A href="#" target="_blank" rel="noopener">best practices</A>, you will ensure that in most cases your queries will be executed without any issue.</P> <P>&nbsp;</P> <P>However, in some cases you might experience some errors of slower queries even if you have properly configured everything. In this case, you can file a support ticket, and synapse support team will analyze your query and handle the issues.</P> <P>&nbsp;</P> <P>It is especially important to include all the information that will help the support team to resolve the issues, for example:</P> <UL> <LI>Workspace name</LI> <LI>Region</LI> <LI>Statement ID</LI> <LI>Error text, query text, Approximate time of the query execution/error, results (if applicable and not contain confidential information)</LI> </UL> <P>The {Statement ID} is very important information that you should include whenever you report an issue. {Statement ID} enables the support team to find most of the information about your query (execution time, duration, error message, etc.), so in most cases it is more valuable than any other information.</P> <P>&nbsp;</P> <P>To avoid delays in the resolution of your support case and going back and forward while trying to get additional information, <STRONG>make sure that you include this statement id in your support ticket</STRONG>.</P> <P>&nbsp;</P> <P>In the following sections, I will explain how to find this information in serverless SQL pool.</P> <P>&nbsp;</P> <H2>Where is my {Statement ID}?</H2> <P>A statement id is information that is returned to the client application whenever you execute a query. Let’s look at a query that is executed using Synapse Studio that returns the following results:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="JovanPop_0-1655813109578.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/382356iB7431BE21E451347/image-size/large?v=v2&amp;px=999" role="button" title="JovanPop_0-1655813109578.png" alt="JovanPop_0-1655813109578.png" /></span></P> <P>&nbsp;</P> <P>Note that the results are displayed in the “Results” tab, but there is another tab called “Messages”. Instead of looking at the results, you should switch to the “Messages” tab by clicking on the “Messages” label:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="JovanPop_1-1655813109593.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/382355i612E2FD0BC40969B/image-size/large?v=v2&amp;px=999" role="button" title="JovanPop_1-1655813109593.png" alt="JovanPop_1-1655813109593.png" /></span></P> <P>&nbsp;</P> <P>In the “Messages” tab, you will be able to see the GUID representing the ID of the statement.</P> <P>&nbsp;</P> <P>If you are using other tools for querying, you can use a similar approach to find the {Statement ID} in the messages. The following figure shows how to find this information when you execute a query in SQL Server Management Studio:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="JovanPop_2-1655813109608.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/382357iCD86348911C0E880/image-size/large?v=v2&amp;px=999" role="button" title="JovanPop_2-1655813109608.png" alt="JovanPop_2-1655813109608.png" /></span></P> <P>&nbsp;</P> <P>If you can execute a query manually via tool, always find the {Statement Id} and send it as a part of support request.</P> <P>&nbsp;</P> <P>The {Statement ID} is sent as an info message using the standard TDS protocol. Therefore, you can get this information even if you are writing code that is accessing serverless pool. You can use <A href="#" target="_blank" rel="noopener">SqlConnection.InfoMessage</A> to get this info in your .Net code. Other libraries and language have similar API for fetching messages.</P> <P>&nbsp;</P> <P>It might be more difficult to find the {Statement ID} if you cannot directly execute the query. For example, Power BI or Analysis services might run the queries in the background to refresh the reports, and you might not be able to see the messages.</P> <P>&nbsp;</P> <P>In that case you should try to find the query and its statement id in the history on serverless pool.</P> <P>&nbsp;</P> <P>First, make sure that you set up the <A href="#" target="_blank" rel="noopener">QPI T-SQL library</A> in your serverless database. This library is a set of useful T-SQL scripts that enable you to find some information in the databases on the serverless SQL pool. Once you execute this script, you will be able to query <STRONG>qpi.query_history</STRONG> view, that returns the recent query executions, their statement ids, duration, data processed etc.</P> <P>The results that this view might return are shown in the following picture:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="JovanPop_3-1655813109646.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/382358i7870E46FD253EB46/image-size/large?v=v2&amp;px=999" role="button" title="JovanPop_3-1655813109646.png" alt="JovanPop_3-1655813109646.png" /></span></P> <P>&nbsp;</P> <P>The value in the request_id column is the value that you should report in a support ticket. This is the easiest way to find the {Statement Id} for the queries that you did not execute directly. Note that at the time of writing this post, the statement id is not shown in the Synapse SQL requests report.</P> <P>&nbsp;</P> <H2>Conclusion</H2> <P>One of the most important pieces of information that you should provide to Synapse support team that is working on the support ticket is the {Statement ID}. This information will enable the support team to quickly find all information about your query and speed up the support case resolution.</P> <P>&nbsp;</P> <P>Make sure that you use this post to find the statement id and include it in the support ticket that you are reporting to ensure that your support case will be more efficiently resolved.</P> Tue, 05 Jul 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/how-to-provide-statement-id-in-the-serverless-sql-pool-support/ba-p/3532926 JovanPop 2022-07-05T15:00:00Z Azure Synapse Analytics June Update 2022 https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-june-update-2022/ba-p/3535086 <DIV id="generatedtoc"> <H2>Azure Synapse Analytics June 2022 Update</H2> </DIV> <P><SPAN data-contrast="auto">Welcome to the June 2022 Azure Synapse Analytics update! This month, you will find information about the Azure Synapse Success by Design playbook and rerunning pipelines with new parameters. Additional general updates and new features in SQL, Synapse Data Explorer, Data Integration, and Machine Learning are also below.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Don’t forget to check out our companion video!</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><IFRAME src="https://www.youtube.com/embed/ZH6Dchwee1s?mute=1&amp;autoplay=1" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <H1>Table of contents</H1> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_0" target="_self" rel="nofollow noopener noreferrer"> General </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_1" target="_self" rel="nofollow noopener noreferrer"> Azure Orbital analytics with Synapse Analytics </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_2" target="_self" rel="nofollow noopener noreferrer"> Azure Synapse Success by Design </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_3" target="_self" rel="nofollow noopener noreferrer"> SQL </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_4" target="_self" rel="nofollow noopener noreferrer"> Result set size limit increase </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_1" target="_self" rel="nofollow noopener noreferrer"> Synapse Data Explorer </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_2" target="_self" rel="nofollow noopener noreferrer"> Web Explorer new homepage </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_4" target="_self" rel="nofollow noopener noreferrer"> Web Explorer sample gallery </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_5" target="_self" rel="nofollow noopener noreferrer"> Web Explorer dashboards drill through capabilities </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_6" target="_self" rel="nofollow noopener noreferrer"> Time Zone settings for Web Explorer </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_7" target="_self" rel="nofollow noopener noreferrer"> Data Integration </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_8" target="_self" rel="nofollow noopener noreferrer"> Fuzzy Join option in Join Transformation </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_9" target="_self" rel="nofollow noopener noreferrer"> Map Data [Generally Available] </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_10" target="_self" rel="nofollow noopener noreferrer"> Rerun pipeline with new parameters </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_11" target="_self" rel="nofollow noopener noreferrer"> User Defined Functions [Generally Available] </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_12" target="_self" rel="nofollow noopener noreferrer"> Machine Learning </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_13" target="_self" rel="nofollow noopener noreferrer"> Distributed Deep Neural Network Training with Horovod and Petastorm [Public Preview]</A></LI> </UL> </LI> </UL> <H2 id="TOCREF_0" aria-level="2"><STRONG><SPAN data-contrast="none">General</SPAN></STRONG></H2> <H3 id="TOCREF_1" aria-level="3"><SPAN data-contrast="none">Azure Orbital analytics with Synapse Analytics</SPAN></H3> <P><SPAN data-contrast="auto">We now offer </SPAN><SPAN data-contrast="none">an </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Orbital analytics sample solution</SPAN></A><SPAN data-contrast="none"> showing an </SPAN><SPAN data-contrast="none">end-to-end implementation of extracting, loading, transforming, and analyzing spaceborne data by using geospatial libraries and AI models with </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse Analytics</SPAN></A><SPAN data-contrast="none">. The sample solution also demonstrates how to integrate geospatial-specific </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Cognitive Services</SPAN></A><SPAN data-contrast="none"> models, AI models from partners, and bring-your-own-data models.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Spaceborne data originating from remote sensors are characterized by high volume, high velocity, and a high degree of variety. To democratize such a unique type of Spatial big data, there is a strong need to build a highly scalable cloud-based solution to generate insights based on artificial intelligence. The solution leverages cloud serverless compute, cost-efficient storage, and high availability across regions to enable an integrated workflow.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Architecture diagram of the workflow with </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">reference architecture</SPAN></A><SPAN data-contrast="none">:</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_0-1656441593118.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384231iD7B1D22FF1C87336/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_0-1656441593118.png" alt="kschaaf_0-1656441593118.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Combining Azure Orbital analytics with Synapse Analytics in this workflow </SPAN><SPAN data-contrast="none">is ideal for the aerospace and aircraft industries. It simplifies the tasks of:</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Raster data ingestion and processing</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="none">Object detection via pre-trained AI models</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><SPAN data-contrast="none">Classification of land masses via AI models</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">Monitoring changes in the environment via AI models</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">Derived datasets from preprocessed imagery sets</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">Vector visualization / small-area consumption</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="11" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">Vector data filtering and cross-data joins</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> </UL> <P>&nbsp;</P> <P><SPAN data-contrast="none">We’ll explore and walk through some of these scenarios in a future blog series. Stay tuned.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To learn more about the Azure orbital analytics workflow, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Spaceborne data analysis with Azure Synapse Analytics</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To learn more about the team behind this solution, visit </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Space</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H3 id="TOCREF_2" aria-level="3"><SPAN data-contrast="none">Azure Synapse Success by Design</SPAN></H3> <P><SPAN data-contrast="auto">Project success is no accident and requires careful planning and execution. The Synapse Analytics' Success by Design playbooks are now available on Microsoft Docs. The </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse proof of concept playbook</SPAN></A><SPAN data-contrast="auto"> provides a guide to scope, design, execute, and evaluate a proof of concept for SQL or Spark workloads. These guides contain best practices from the most challenging and complex solution implementations incorporating Azure Synapse.</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">If you're working on implementing a Synapse Analytics solution, plug the Implementation Success method right into your existing project plan or use it as a guide. You will find key checkpoints at every step of the implementation process to validate the end-to-end design for workspaces, security, data integration, SQL, and Spark pools.</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about the Azure Synapse proof of concept playbook, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Success by Design</SPAN></A><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H2 id="TOCREF_3" aria-level="2"><STRONG><SPAN data-contrast="none">SQL</SPAN></STRONG></H2> <H3 id="TOCREF_4" aria-level="3"><SPAN data-contrast="none">Result set size limit increase</SPAN></H3> <P><SPAN data-contrast="auto">We know that you turn to Azure Synapse Analytics to work with large amounts of data. With that in mind, the maximum size of query result sets in Serverless SQL pools has been increased from 200GB to 400GB. This limit is shared between concurrent queries.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about this size limit increase and other constraints, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Self-help for serverless SQL pool</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H2 id="TOCREF_1" aria-level="2"><STRONG><SPAN data-contrast="none">Synapse Data Explorer</SPAN></STRONG></H2> <H3 id="TOCREF_2" aria-level="3"><SPAN data-contrast="none">Web Explorer new homepage</SPAN></H3> <P><SPAN data-contrast="auto">The new Synapse Web Explorer homepage makes it even easier to get started with Synapse Web Explorer. The </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Web Explorer homepage</SPAN></A><SPAN data-contrast="auto"> now includes the following sections:&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="·" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;·&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><I><SPAN data-contrast="auto">Get started</SPAN></I><STRONG><SPAN data-contrast="auto"> – </SPAN></STRONG><SPAN data-contrast="auto">Sample gallery offering example queries and dashboards for popular Synapse Data Explorer use cases.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></LI> <LI data-leveltext="·" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;·&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><I><SPAN data-contrast="auto">Recommended </SPAN></I><STRONG><SPAN data-contrast="auto">– </SPAN></STRONG><SPAN data-contrast="auto">Popular learning modules designed to help you master Synapse Web Explorer and KQL.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></LI> <LI data-leveltext="·" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;·&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><I><SPAN data-contrast="auto">Documentation</SPAN></I><STRONG><SPAN data-contrast="auto"> – </SPAN></STRONG><SPAN data-contrast="auto">Synapse Web Explorer basic and advanced documentation.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_1-1656441593120.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384233i7023EE4532499DEA/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_1-1656441593120.png" alt="kschaaf_1-1656441593120.png" /></span></P> <H3 id="TOCREF_3">&nbsp;</H3> <H3 id="TOCREF_4"><SPAN><BR /></SPAN><SPAN data-contrast="none">Web Explorer sample gallery</SPAN></H3> <P><SPAN data-contrast="auto">A great way to learn about a product is to see how it is being used by others. The Web Explorer sample gallery provides end-to-end samples of how customers leverage Synapse Data Explorer popular use cases such as Logs Data, Metrics Data, IoT data and Basic big data examples. Each sample includes the dataset, well-documented queries, and a sample dashboard.&nbsp;&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Navigate to the sample gallery from the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Web Explorer homepage</SPAN></A><SPAN data-contrast="auto">. From the homepage’s </SPAN><I><SPAN data-contrast="auto">Get Started</SPAN></I><SPAN data-contrast="auto"> section, go to </SPAN><I><SPAN data-contrast="auto">Explore sample data with KQL </SPAN></I><SPAN data-contrast="auto">to access the sample queries or go to </SPAN><I><SPAN data-contrast="auto">Explore sample dashboards </SPAN></I><SPAN data-contrast="auto">to open the sample dashboards.&nbsp;&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_2-1656441593122.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384232iC69DFDBF471A07CE/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_2-1656441593122.png" alt="kschaaf_2-1656441593122.png" /></span></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To access the sample gallery, you need </SPAN><SPAN data-contrast="none">either an Azure active directory (AAD) user identity or a </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Microsoft account (MSA)</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about the sample gallery, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-explorer-blog/azure-data-explorer-in-60-minutes-with-the-new-samples-gallery/ba-p/3447552" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Data Explorer in 60 minutes with the new samples gallery</SPAN></A><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H3 id="TOCREF_5"><SPAN data-contrast="none">Web Explorer dashboards drill through capabilities</SPAN></H3> <P><SPAN data-contrast="none">You can now add d</SPAN><SPAN data-contrast="auto">rill through capabilities</SPAN><SPAN data-contrast="none"> to your Synapse Web Explorer dashboards. The new drill through capabilities allow you to easily jump back and forth between dashboard pages. This is made possible by using a contextual filter to connect your dashboards. Defining these contextual drill throughs is done by editing the visual interactions of the selected tile in your dashboard.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_3-1656441593123.png" style="width: 456px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384234i4450E3D5D63D1A1B/image-dimensions/456x400?v=v2" width="456" height="400" role="button" title="kschaaf_3-1656441593123.png" alt="kschaaf_3-1656441593123.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">To learn more about drill through capabilities, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Use drillthroughs as dashboard parameters</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H3 id="TOCREF_6" aria-level="3"><SPAN data-contrast="none">Time Zone settings for Web Explorer</SPAN></H3> <P><SPAN data-contrast="none">Being able to display data in different time zones is very powerful. You can now decide to view the data in UTC time, your local time zone, or the time zone of the monitored device/machine. The Time Zone settings of the Web Explorer now apply to both the Query results and to the Dashboard. By changing the time zone, the dashboards will be automatically refreshed to present the data with the selected time zone.&nbsp;</SPAN><SPAN data-contrast="none">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_4-1656441593124.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384235iB2699A84627C5718/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_4-1656441593124.png" alt="kschaaf_4-1656441593124.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">For more information on time zone settings, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Change datetime to specific time zone</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H2 id="TOCREF_7" aria-level="2"><STRONG><SPAN data-contrast="none">Data Integration</SPAN></STRONG></H2> <H3 id="TOCREF_8" aria-level="3"><SPAN data-contrast="none">Fuzzy Join option in Join Transformation</SPAN></H3> <P><SPAN data-contrast="auto">Fuzzy matching with a sliding similarity score option has been added to the Join transformation in Mapping Data Flows. You can create inner and outer joins on data values that are similar rather than exact matches! Previously, you would have had to use an exact match. The sliding scale value goes from 60% to 100%, making it easy to adjust the similarity threshold of the match.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_5-1656441593126.png" style="width: 605px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384236i93353A068EC7D072/image-dimensions/605x379?v=v2" width="605" height="379" role="button" title="kschaaf_5-1656441593126.png" alt="kschaaf_5-1656441593126.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">For learn more about fuzzy joins, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Join transformation in mapping data flow</SPAN></A><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P aria-level="3"><SPAN data-ccp-props="{&quot;335559738&quot;:40,&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H3 id="TOCREF_9" aria-level="3"><SPAN data-contrast="none">Map Data [Generally Available]</SPAN></H3> <P><SPAN data-contrast="auto">We’re excited to announce that the Map Data tool is now Generally Available. The Map Data tool is a guided process to help you create ETL mappings and mapping data flows from your source data to Synapse without writing code.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_6-1656441593128.png" style="width: 518px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384238iB2CB370DAE36AB71/image-dimensions/518x298?v=v2" width="518" height="298" role="button" title="kschaaf_6-1656441593128.png" alt="kschaaf_6-1656441593128.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">For learn more about Map Data, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Map Data in Azure Synapse Analytics</SPAN></A><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P aria-level="3"><SPAN data-ccp-props="{&quot;335559738&quot;:40,&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H3 id="TOCREF_10" aria-level="3"><SPAN data-contrast="none">Rerun pipeline with new parameters</SPAN></H3> <P><SPAN data-contrast="none">You can now change pipeline parameters when re-running a pipeline from the Monitoring page without having to return to the</SPAN><SPAN data-contrast="none"> pipeline editor</SPAN><SPAN data-contrast="none">. After running a pipeline with new parameters, y</SPAN><SPAN data-contrast="none">ou can easily monitor the new run against the old ones without having to toggle between pages.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Note that by re-running a pipeline with new parameters, it will be considered a new pipeline run and will not show under re-run groupings.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_7-1656441593129.png" style="width: 577px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384239iEE76B34112000728/image-dimensions/577x391?v=v2" width="577" height="391" role="button" title="kschaaf_7-1656441593129.png" alt="kschaaf_7-1656441593129.png" /></span></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To learn more about rerunning pipelines with new parameters, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Rerun pipelines and activities</SPAN></A><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P aria-level="3"><SPAN data-ccp-props="{&quot;335559738&quot;:40,&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H3 id="TOCREF_11" aria-level="3"><SPAN data-contrast="none">User Defined Functions [Generally Available]</SPAN></H3> <P><SPAN data-contrast="auto">We’re excited to announce that user defined functions (UDFs) are now Generally Available. </SPAN><SPAN data-contrast="auto">With user-defined functions, you can create customized expressions that can be reused across multiple mapping data flows. You no longer have to use </SPAN><SPAN data-contrast="none">the same string manipulation, math calculations, or other complex </SPAN><SPAN data-contrast="auto">logic several times. User-defined functions will be grouped in libraries to help developers group common sets of functions.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_8-1656441593130.png" style="width: 547px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/384237i8AB3C7F252C20DA1/image-dimensions/547x286?v=v2" width="547" height="286" role="button" title="kschaaf_8-1656441593130.png" alt="kschaaf_8-1656441593130.png" /></span></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about user defined functions, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-factory-blog/introducing-user-defined-functions-preview-for-mapping-data/ba-p/3414628" target="_blank" rel="noopener"><SPAN data-contrast="none">User defined functions in mapping data flows</SPAN></A><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <H2 id="TOCREF_12" aria-level="2"><STRONG><SPAN data-contrast="none">Machine Learning</SPAN></STRONG></H2> <H3 id="TOCREF_13" aria-level="3"><SPAN data-contrast="none">Distributed Deep Neural Network Training with Horovod and Petastorm [Public Preview]</SPAN></H3> <P><SPAN data-contrast="auto">To simplify the process for creating and managing GPU-accelerated pools, Azure Synapse takes care of pre-installing low-level libraries and setting up all the complex networking requirements between compute nodes. This integration allows users to get started with GPU- accelerated pools within just a few minutes.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Now, Azure Synapse Analytics provides built-in support for deep learning infrastructure.&nbsp; The Azure Synapse Analytics runtime for Apache Spark 3.1 and 3.2 now includes support for the most common deep learning libraries like TensorFlow and PyTorch. The Azure Synapse runtime also includes supporting libraries like Petastorm and Horovod, which are commonly used for distributed training. This feature is currently available in Public Preview.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about how to leverage these libraries within your Azure Synapse Analytics GPU-accelerated pools, read the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Deep learning</SPAN></A><SPAN data-contrast="auto"> tutorials.</SPAN><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;335559739&quot;:0}">&nbsp;</SPAN></P> Wed, 29 Jun 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-june-update-2022/ba-p/3535086 ryanmajidi 2022-06-29T15:00:00Z CICD Automation in Synapse Analytics: taking advantage of custom parameters in Workspace Templates https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/cicd-automation-in-synapse-analytics-taking-advantage-of-custom/ba-p/3513546 <H2>Introduction</H2> <P>&nbsp;</P> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="f6q8r-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="f6q8r-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="f6q8r-0-0">Azure Synapse Analytics is an integrated analytics platform, which combines data warehousing, big data analytics, data integration, and visualization into a single environment</SPAN></SPAN><SPAN data-offset-key="f6q8r-1-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="f6q8r-2-0">Azure Synapse Studio is the primary tool to interact with the many components in the platform, allowing a wide range of activities against your data</SPAN></SPAN><SPAN data-offset-key="f6q8r-3-0">.</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="vama-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="vama-0-0"><SPAN data-offset-key="vama-0-0">&nbsp;</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="dbaa6-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="dbaa6-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="dbaa6-0-0">When working </SPAN></SPAN><SPAN class="adverb"><SPAN data-offset-key="dbaa6-1-0">collaboratively</SPAN></SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="dbaa6-2-0"> in Synapse Studio, it </SPAN></SPAN><SPAN class="passivevoice"><SPAN data-offset-key="dbaa6-3-0">is recommended</SPAN></SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="dbaa6-4-0"> that you <A href="#" target="_blank" rel="noopener">integrate your Synapse Workspace with your Git repository</A> to take advantage of source control, as it will allow developers to collaborate on code and track changes</SPAN></SPAN><SPAN data-offset-key="dbaa6-5-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="dbaa6-6-0">Working in a multi-environment scenario that requires the implementation of Continuous Integration and Continuous Delivery processes is also an important reason for integrating your Synapse Workspace with a source control system</SPAN></SPAN><SPAN data-offset-key="dbaa6-7-0">.</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="734fv-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="734fv-0-0"><SPAN data-offset-key="734fv-0-0">&nbsp;</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="93od2-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="93od2-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="93od2-0-0">Automating the delivery of your Synapse artifacts (pipelines,notebooks,sql scripts,etc) across multi-environments scenarios need these artifacts to fit the different characteristics of the referenced resources in these environments</SPAN></SPAN><SPAN data-offset-key="93od2-1-0">. Here’s an example: a Linked Service that is using an Azure Key Vault (AKV) to get a connection string from a Secret. Different environments will use different AKVs and different Secrets. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="93od2-2-0">When delivering this Linked Service across many environments you want that linked service to refer to the appropriate AKV and Secret in each environment</SPAN></SPAN><SPAN data-offset-key="93od2-3-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="93od2-4-0">You can achieve this by taking advantage of <A href="#" target="_blank" rel="noopener">custom template parameterization</A> in your Synapse CICD lifecycle</SPAN></SPAN><SPAN data-offset-key="93od2-5-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="93od2-6-0">The goal of this article is to show how you can use custom parameters in your Synapse Workspace templates to ease the delivery of your code in a multi-environment scenario</SPAN></SPAN></DIV> </DIV> <P>&nbsp;</P> <H2>Before we start…<BR /><BR /></H2> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="8pren-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="8pren-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="8pren-0-0">The Continuous Integration process in Synapse Workspace starts from the moment you integrate your Workspace with your Git repository, and you begin writing, testing, and reviewing your code, building a shared codebase with your team members</SPAN></SPAN><SPAN data-offset-key="8pren-1-0">. </SPAN><SPAN class="hardreadability"><SPAN data-offset-key="8pren-2-0">The culmination of this process happens when you <A href="#" target="_blank" rel="noopener">Publish your code</A> and generate the ARM templates into your Git repository</SPAN></SPAN><SPAN data-offset-key="8pren-3-0">.</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="f0une-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="f0une-0-0"><SPAN data-offset-key="f0une-0-0">&nbsp;</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="6i23a-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="6i23a-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-0-0">These ARM templates will be the input for your Continuous Delivery process, as they contain all the workspace artifacts that you want to deploy into a target environment</SPAN></SPAN><SPAN data-offset-key="6i23a-1-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-2-0">Although these ARM templates contain all the workspace artifacts and their corresponding properties, not all artifacts’ properties </SPAN></SPAN><SPAN class="passivevoice"><SPAN data-offset-key="6i23a-3-0">are exposed</SPAN></SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-4-0"> and parameterized by default</SPAN></SPAN><SPAN data-offset-key="6i23a-5-0">. </SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-6-0">The default parameter template </SPAN></SPAN><SPAN class="passivevoice"><SPAN data-offset-key="6i23a-7-0">is limited</SPAN></SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-8-0"> to a few artifact properties, and you will observe that as you create different artifacts in your workspace you will need other artifacts properties to </SPAN></SPAN><SPAN class="passivevoice"><SPAN data-offset-key="6i23a-9-0">be exposed</SPAN></SPAN><SPAN class="veryhardreadability"><SPAN data-offset-key="6i23a-10-0"> rather than the default</SPAN></SPAN><SPAN data-offset-key="6i23a-11-0">. This is where the custom template parameterization comes in.</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="8vq38-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="8vq38-0-0"><SPAN data-offset-key="8vq38-0-0">&nbsp;</SPAN></DIV> </DIV> <DIV class="" data-block="true" data-editor="b487r" data-offset-key="ftdn4-0-0"> <DIV class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="ftdn4-0-0"><SPAN class="veryhardreadability"><SPAN data-offset-key="ftdn4-0-0">This article outlines how to use a custom parameters template to override non-default properties in your Synapse Workspace ARM templates enhancing the Continuous Delivery process for your Synapse Workspace artifacts</SPAN></SPAN><SPAN data-offset-key="ftdn4-1-0">.</SPAN></DIV> </DIV> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <H2>Synapse Workspace ARM Templates</H2> <P>&nbsp;</P> <P>When you integrate your Synapse Workspace with your Git repository, you need to define the organizational structure of your Source Control System (Organization-&gt;Project-&gt;Repository) , your workspace shared codebase folder (collaboration branch) and workspace ARM templates folder (publish branch).</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_0-1655333097620.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380745i89CED0D9C59E32D1/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_0-1655333097620.png" alt="RuiCunha_0-1655333097620.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P><SPAN class="hardreadability"><SPAN data-offset-key="8pren-0-0"> Once you integrate your workspace with your Git repository, you will no longer be authoring your code against the Synapse service</SPAN></SPAN><SPAN data-offset-key="8pren-1-0">. </SPAN><SPAN class="hardreadability"><SPAN data-offset-key="8pren-2-0">Instead, all the changes will be first committed to your Git repository before getting published in Synapse Service (Live Mode)</SPAN></SPAN><SPAN data-offset-key="8pren-3-0">.</SPAN><BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_1-1655333097673.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380747iD17336A2FD52A5EA/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_1-1655333097673.png" alt="RuiCunha_1-1655333097673.png" /></span></P> <P><FONT size="2">The Continuous Integration Process in Synapse Workspace</FONT><BR /><BR /></P> <P>The Publish operation is divided in two stages: a first stage where all the pending changes from your Git collaboration branch are stored in your workspace (Live Mode); and a second stage where the workspace ARM templates are generated and saved in the workspace publish branch. These <STRONG>two</STRONG> ARM templates represent the outcome of the Continuous Integration process in your Synapse Workspace:</P> <P><STRONG>TemplateForWorkspace.json</STRONG> is the ARM template containing all the workspace artifacts and resources</P> <P><STRONG>TemplateParametersForWorkspace.json</STRONG> is the ARM template containing <STRONG>only the artifacts parameters.</STRONG></P> <P><STRONG>&nbsp;</STRONG></P> <P>&nbsp;</P> <H3>The default parameters template</H3> <P>&nbsp;</P> <P>After integrating your Synapse Workspace, your first Publish will generate a TemplateParametersForWorkspace.json file containing a global parameter for your workspace name and a parameter for each workspace default linked service: the default SQL Server and the default Storage account.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_2-1655333097679.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380746i55EC83C4B03EF4FA/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_2-1655333097679.png" alt="RuiCunha_2-1655333097679.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Where are these parameters coming from?<BR /><BR /></P> <P>When in Synapse Studio, go to the "Manage Hub" -&gt; select "Linked Services" and mouse over one of the workspace default linked services (in this example below I'm selecting the default sql server) and select “Code” {}<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_3-1655333097681.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380749i1BD1BCDAE6534A8C/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_3-1655333097681.png" alt="RuiCunha_3-1655333097681.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>You can see these properties highlighted below that are being exposed by the Workspace default parameter template.<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_4-1655333097684.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380748iD5FC0CCE44BCD424/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_4-1655333097684.png" alt="RuiCunha_4-1655333097684.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_5-1655333097685.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380750i93E595D5C4C181E6/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_5-1655333097685.png" alt="RuiCunha_5-1655333097685.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Now let’s create a new Linked Service, using the Azure Key Vault connector, and publish the pending changes to generate the new ARM templates.<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_6-1655333097687.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380751i40C077A7BEB6CBD1/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_6-1655333097687.png" alt="RuiCunha_6-1655333097687.png" /></span></P> <P>&nbsp;</P> <P>Check the TemplateParametersForWorkspace.json in your publish branch to confirm that the new AKV linked service "baseUrl" property is also being exposed by the default parameter template.<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_7-1655333097691.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380753i427E21ACF87BF33E/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_7-1655333097691.png" alt="RuiCunha_7-1655333097691.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_8-1655333097693.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380752i1543F9F2EB78B7D4/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_8-1655333097693.png" alt="RuiCunha_8-1655333097693.png" /></span></P> <P>&nbsp;</P> <P>Now let’s create and publish a Notebook attached to an existing Spark Pool.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_9-1655333097695.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380754i4C76016951D3C577/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_9-1655333097695.png" alt="RuiCunha_9-1655333097695.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Check the TemplateParametersForWorkspace.json in your publish branch. No sign of any notebook property,right?</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_10-1655333097700.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380755i7707AD9920F3AE47/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_10-1655333097700.png" alt="RuiCunha_10-1655333097700.png" /></span></P> <P>&nbsp;</P> <P>But if you check the TemplateForWorkspace.json in your publish branch, you will find several notebook properties!! Here’s a clear example of an artifact whose properties are not exposed by the default workspace parameters template.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_11-1655333097704.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380756iF6DDEA3F33DA8A9F/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_11-1655333097704.png" alt="RuiCunha_11-1655333097704.png" /></span></P> <P>&nbsp;</P> <P>Let's use a different kind of artifact, a Dataset, and see if the default template will expose its properties.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_12-1655333097706.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380757i79633C19154A60A5/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_12-1655333097706.png" alt="RuiCunha_12-1655333097706.png" /></span></P> <P>&nbsp;</P> <P>Again, no sign of these Dataset properties in the parameters file:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_13-1655333097710.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380758iE4B59078C8B5DF2A/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_13-1655333097710.png" alt="RuiCunha_13-1655333097710.png" /></span></P> <P>&nbsp;</P> <P>Although these Datasets are part of the main template file with several properties associated:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_14-1655333097719.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380759i74B1E948465C4EE3/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_14-1655333097719.png" alt="RuiCunha_14-1655333097719.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <H3>Using a customized parameters template</H3> <P>&nbsp;</P> <P>The use case for using a customized parameters template in Synapse is simple: when you want to automate your CICD process in Synapse and you need to override any artifact property that is not parameterized by the default parameters template.</P> <P>&nbsp;</P> <H5>How does it work?</H5> <P>&nbsp;</P> <P>After publishing your pending changes from the collaboration branch into Synapse Service (Live Mode), Synapse will verify if there is any <A href="#" target="_blank" rel="noopener">custom template file</A> stored in the <STRONG>root folde</STRONG>r of your collaboration branch with this <STRONG>exact name</STRONG> “<STRONG>template-parameters-definition.json</STRONG>”. If this file exists, Synapse will use its configuration to generate the ARM template parameters; if it does not exist, it will use the default parameters template.</P> <P>&nbsp;</P> <H5>Creating your custom parameters template</H5> <P>&nbsp;</P> <P>From your Devops collaboration branch, hit the “More Actions” button and then select + New -&gt; File to create a new file <STRONG>in the root folder</STRONG> of your collaboration branch.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_15-1655333097723.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380761iCA42979054E057D9/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_15-1655333097723.png" alt="RuiCunha_15-1655333097723.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Important</STRONG>: Create a new file with this exact name: <STRONG>template-parameters-definition.json</STRONG></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_16-1655333097726.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380760i00AB9E861AAB2289/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_16-1655333097726.png" alt="RuiCunha_16-1655333097726.png" /></span></P> <P>&nbsp;</P> <P>Hit the “Create” button and copy the parameters template definition JSON example from Microsoft Public documents: <A href="#" target="_blank" rel="noopener">Continuous integration &amp; delivery in Azure Synapse Analytics - Azure Synapse Analytics | Microsoft Docs</A></P> <P>&nbsp;</P> <P>Paste the JSON content into the new <STRONG>template-parameters-definition.json&nbsp;</STRONG>file. Don’t forget to select “Commit” to save your changes.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_17-1655333097734.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380762i0909A49498BA2CCE/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_17-1655333097734.png" alt="RuiCunha_17-1655333097734.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Now that we have saved the custom template file, it’s time to generate the new Synapse Workspace ARM templates.</P> <P>Switch to Synapse Studio and do a minor change in your code to force a new commit, publish this change in the Live Service to generate the new ARM template files.</P> <P>Once the ARM templates get generated, check the TemplateParameterForWorskspace.json arm template in your publish branch. This file content will now look much different from the original one, as you have now exposed more properties to parameterize.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_18-1655333097744.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380765i0261014B16572206/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_18-1655333097744.png" alt="RuiCunha_18-1655333097744.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>You may ask: If we have published the Storage Account and the Synapse SQL pool datasets, why the properties for the latest are missing in the TemplateParameterForWorskspace.json?</P> <P>&nbsp;</P> <P>Let’s take a look at the template-parameters-definition.json file and check the /datasets section:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_19-1655333097746.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380763iCD529DC9D4945C8A/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_19-1655333097746.png" alt="RuiCunha_19-1655333097746.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>We are exposing any key-value pairs that are included under the “properties” -&gt; “typeProperties” object.</P> <P>&nbsp;</P> <P>Let’s analyze the JSON code associated with each dataset.</P> <P>Starting with the Storage Account dataset:</P> <P>we have four key-value pairs listed under the “typeProperties” object:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_20-1655333097750.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380764i8C9248F4E0656E1B/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_20-1655333097750.png" alt="RuiCunha_20-1655333097750.png" /></span></P> <P>&nbsp;</P> <P>Looking at the TemplateParameterForWorskspace.json file, we confirm the presence of these properties.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_21-1655333097752.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380766iA694CA2B066B1769/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_21-1655333097752.png" alt="RuiCunha_21-1655333097752.png" /></span></P> <P>&nbsp;</P> <P>Now let's look at the Synapse SQL&nbsp;pool dataset JSON.</P> <P>Since there are no key-value pairs under the “typeProperties” object, no properties will be exposed in the ARM template to parameterize.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_22-1655333097756.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380767i338BB93386386C25/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_22-1655333097756.png" alt="RuiCunha_22-1655333097756.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <H3>Example: Using a custom parameter template to attach a Notebook to a different Spark Pool</H3> <P>&nbsp;</P> <P>Microsoft <A title="Best practices for CI/CD" href="#" target="_blank" rel="noopener">strongly recommends</A> that you prepare your pools before migrating the workspace artifacts, making sure you use the same name for your pools across your environments. In some circumstances, you may need to attach your artifacts to a different pool in your target environment. Using a custom parameter template can help you achieve this goal.</P> <P>&nbsp;</P> <P>In this example, I’m going to show how you can take advantage of custom parameterization in your parameters template to attach a Notebook to a different Spark Pool, when deploying this Notebook to a target environment hosting a Spark pool with a different name.</P> <P>So here’s the case where you have two environments each one hosting a Spark Pool with different names.</P> <P>&nbsp;</P> <TABLE class=" lia-align-center"> <TBODY> <TR style="background-color: #ffffff;"> <TD width="354"> <P><STRONG>DEV Environment</STRONG></P> </TD> <TD width="354"> <P><STRONG>UAT Environment</STRONG></P> </TD> </TR> <TR> <TD width="354" class="lia-align-center" style="width: 354px; vertical-align: top;"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="RuiCunha_23-1655333097757.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380768i8743CD6A0FBB2E88/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_23-1655333097757.png" alt="RuiCunha_23-1655333097757.png" /></span> <P>&nbsp;</P> </TD> <TD width="354" class="lia-align-center" style="width: 354px; vertical-align: top;"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_24-1655333097759.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380769i14A28F15ED63834E/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_24-1655333097759.png" alt="RuiCunha_24-1655333097759.png" /></span> <P>&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>Here’s an example of a Notebook in the DEV environment that is attached to a Spark Pool named “mysparkpooldev”.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_25-1655333097762.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380770i427F63D2EC755721/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_25-1655333097762.png" alt="RuiCunha_25-1655333097762.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Taking a closer look at the Notebook JSON code, there are multiple properties where this Spark Pool is being referenced.<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_26-1655333097770.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380771iB51C39E4836C12AD/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_26-1655333097770.png" alt="RuiCunha_26-1655333097770.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Now we need to change the template definition file (template-parameters-definition.json), and find the Microsoft.Synapse/workspaces/notebooks section to expose these additional properties. You can find <STRONG><FONT color="#0000FF">highlighted</FONT> </STRONG>below, the code that you need to add to this section to expose these properties.</P> <P>&nbsp;</P> <P>"Microsoft.Synapse/workspaces/notebooks":&nbsp;{</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"properties":&nbsp;{</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"bigDataPool":&nbsp;{</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"referenceName":&nbsp;"="</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<STRONG><FONT color="#0000FF">&nbsp;&nbsp;"metadata":&nbsp;{</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"a365ComputeOptions":&nbsp;{</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"id":&nbsp;"=",</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp; "name":&nbsp;"=",</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"endpoint":&nbsp;"="</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</FONT></STRONG></P> <P><STRONG><FONT color="#0000FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</FONT></STRONG></P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</P> <P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</P> <P>&nbsp;</P> <P>Don’t forget to select "<STRONG>Commit</STRONG>" to save your changes.</P> <P>&nbsp;</P> <P>Switch now to Synapse Studio, and make sure you make a minor change in your notebook to force a commit and publish your changes. This will generate the ARM templates based on the new template definition file.</P> <P>Once the template generation is finished, check the TemplateParametersForWorkspace.json in your workspace publish branch to confirm that the new notebook parameters are now being exposed.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_27-1655333097776.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380774i9159802C41A0B8A7/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_27-1655333097776.png" alt="RuiCunha_27-1655333097776.png" /></span></P> <P>&nbsp;</P> <P>Once you confirm that the necessary properties are being exposed,&nbsp; you can config the Workspace Deployment task in your Release Pipeline and add these new parameters in the “OverrideParameters” section.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_28-1655333097780.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380773i57DDAF4666741032/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_28-1655333097780.png" alt="RuiCunha_28-1655333097780.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>As an example, I’m overriding this parameter “NotebookA_properties_metadata_a365ComputeOptions_id” using the target Spark Pool resourceURi:</P> <P>/subscriptions/&lt;target_workspace_subscription&gt;/resourceGroups/&lt;target_workspace_RG&gt;/providers/Microsoft.Synapse/workspaces/&lt;target_workspace_name&gt;/bigDataPools/&lt;target_spark_pool&gt;</P> <P>&nbsp;</P> <P>After executing your Release Pipeline in Azure DevOps, go to your target Synapse Workspace and open the Notebook to confirm that it is now attached to a Spark pool with a different name.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_29-1655333097782.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380772i91E52BEA5ECC4842/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_29-1655333097782.png" alt="RuiCunha_29-1655333097782.png" /></span></P> <P>&nbsp;</P> <H3>Parameter renaming</H3> <P>&nbsp;</P> <P>To simplify the parameter overriding operation and code maintenance, you can take advantage of a custom parameters template to provide shorter names to your parameters.</P> <P>Let’s take this parameter name as example: <STRONG>NotebookA_properties_metadata_a365ComputeOptions_id</STRONG>. Lengthy name, right?</P> <P>Let's make this parameter name shorter, &nbsp;like “<STRONG>NotebookA_meta_id</STRONG>”.</P> <P>&nbsp;</P> <P>You just need to edit the Notebooks section in your template-parameters-definition.json and use the custom parameter syntax as explained <A href="#" target="_blank" rel="noopener">here</A>.</P> <P>&nbsp;</P> <P>Use the format <STRONG>&lt;action&gt;:&lt;name&gt;:&lt;stype&gt;</STRONG></P> <P><STRONG>&lt;action&gt; -&gt;</STRONG>&nbsp;we are using the “=” character to keep the current current value as the default value for the parameter.</P> <P><STRONG>&lt;name&gt; -&gt;</STRONG>&nbsp;we are using “-“ character (because we don’t want to keep the default name) followed by the new name.</P> <P><STRONG>&lt;stype&gt; -&gt;</STRONG> we don’t want to change the default type, so we are omitting this value (by default the parameter type is a string).</P> <P>&nbsp;</P> <P>Here’s how the Notebook section will look like:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_30-1655333097784.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380775i2C32DC7E62D0916B/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_30-1655333097784.png" alt="RuiCunha_30-1655333097784.png" /></span></P> <P>&nbsp;</P> <P>Now if you switch back to Synapse Studio and publish&nbsp;any pending changes from your collaboration branch to generate the new ARM templates, you will see that this Notebook parameter has been renamed from “NotebookA_properties_metadata_a365ComputeOptions_id” to “NotebookA_meta_id”.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="RuiCunha_31-1655333097789.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/380776iB0AC213262E78008/image-size/medium?v=v2&amp;px=400" role="button" title="RuiCunha_31-1655333097789.png" alt="RuiCunha_31-1655333097789.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <H4>Known Limitations</H4> <P>&nbsp;</P> <P>At the time of this writing, Synapse will fail to generate the ARM templates if they exceed the 20MB limit each.</P> <P>If you are experiencing this limitation and failing to generate these ARM templates during the publish operation, you can evaluate if by using a custom parameters template and renaming you parameters to use shorter parameter names will decrease the ARM file size and allow the ARM template generation.</P> <P>&nbsp;</P> <H2>Conclusion</H2> <P>&nbsp;</P> <P>When using automated CI/CD in Azure Synapse Analytics, users can take advantage of custom parameters to extend the capabilities of the default Workspace template, allowing the exposure and the overriding of any artifact property that is not parameterized by default.</P> <P>&nbsp;</P> <H3 id="recommended-content" class="title is-3 margin-top-sm">Recommended content<BR /><BR /></H3> <P><A href="#" target="_blank" rel="noopener">Source control in Synapse Studio - Azure Synapse Analytics | Microsoft Docs</A></P> <P>Learn how to configure source control in Synapse Studio</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Create custom parameters in the workspace template</A></P> <P>Learn how to use custom parameters in Synapse CICD</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Best practices for CI/CD in Azure Synapse Analytics</A></P> <P><SPAN>If you're using Git integration with your Azure Synapse workspace&nbsp;we recommend these best practices</SPAN></P> <P>&nbsp;</P> Tue, 21 Jun 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/cicd-automation-in-synapse-analytics-taking-advantage-of-custom/ba-p/3513546 RuiCunha 2022-06-21T15:00:00Z Join us for our first Azure Synapse Influencers Ask the Experts session https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/join-us-for-our-first-azure-synapse-influencers-ask-the-experts/ba-p/3507713 <DIV style="text-align: center;"><IFRAME src="https://www.youtube.com/embed/w2XHfALv_UM" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></DIV> <P><SPAN data-contrast="auto">I’m pleased to share that </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">registration is now open</SPAN></A><SPAN data-contrast="auto"> for our first-ever Azure Synapse Influencers event on </SPAN><STRONG><SPAN data-contrast="auto">June 29, 2022 (10AM to 11AM Pacific Time)</SPAN></STRONG><SPAN data-contrast="auto">.</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">This Ask the Experts session will feature Principal Program Managers Mark Kromer and Sunil Sabat</SPAN><STRONG><SPAN data-contrast="auto">,</SPAN></STRONG><SPAN data-contrast="auto"> who will answer your questions on all things </SPAN><STRONG><SPAN data-contrast="auto">Data Integration in Azure Synapse</SPAN></STRONG><SPAN data-contrast="auto">, covering Azure Data Factory, Azure Synapse Pipelines, Azure Synapse Link, and SSIS. Event attendees are encouraged to ask questions during the event or submit questions in advance at </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">signup</SPAN></A><SPAN data-contrast="auto">.</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Registration is free for all Azure Synapse Influencers, and if you’re not an influencer yet, you can sign up today at </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">https://aka.ms/synapseinfluencers</SPAN></A><SPAN data-contrast="auto">.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">I’d like to thank everyone who’s already joined the program and helped support the community by sharing Synapse-related announcements and educational content. The program has far surpassed our expectations for participation, and we’ve already welcomed more than 400 influencers! Please remember to use the hashtags </SPAN><STRONG><SPAN data-contrast="auto">#AzureSynapse</SPAN></STRONG><SPAN data-contrast="auto"> and </SPAN><STRONG><SPAN data-contrast="auto">#SynapseInfluencers</SPAN></STRONG><SPAN data-contrast="auto"> in your posts to ensure other influencers can find and amplify your content.</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">We’re excited to build on the momentum of the program with this Ask the Experts (ATE) event. This will be the first in a series of ATE events covering a wide range of topics and featuring engineers and guest speakers from across the Synapse community.</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-center"><A href="#" target="_blank" rel="noopener"><STRONG><SPAN data-contrast="none">Register today &gt;</SPAN></STRONG></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> Wed, 15 Jun 2022 18:40:14 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/join-us-for-our-first-azure-synapse-influencers-ask-the-experts/ba-p/3507713 ryanmajidi 2022-06-15T18:40:14Z Retrieve Azure Synapse role-based access control (RBAC) Information using PowerShell https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/retrieve-azure-synapse-role-based-access-control-rbac/ba-p/3466419 <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Azure Synapse Analytics</A>&nbsp;is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, manage and serve data for immediate BI and machine learning needs.</P> <P>Synapse RBAC extends the capabilities of Azure RBAC for Synapse workspaces and their content.</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Synapse RBAC</A> is used to manage who can:</P> <UL> <LI>Publish code artifacts and list or access published code artifacts,</LI> <LI>Execute code on Apache Spark pools and Integration runtimes,</LI> <LI>Access linked (data) services protected by credentials</LI> <LI>Monitor or cancel job execution, review job output, and execution logs.</LI> </UL> <P><A href="#" target="_blank" rel="noopener">Azure Synapse RBAC</A> has built-in roles and scopes that helps to manage permissions in Azure Synapse Analytics -&nbsp;</P> <P>&nbsp;</P> <DIV style="direction: ltr;"> <TABLE style="direction: ltr; border-collapse: collapse; border: 1pt solid #A3A3A3;" title="" border="1" summary="" cellspacing="0" cellpadding="0"> <TBODY> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; text-align: center;"><SPAN>Role</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; text-align: center;"><SPAN>Permissions</SPAN></P> </TD> <TD style="vertical-align: top; width: 1.702in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; text-align: center;"><SPAN>Scopes</SPAN></P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse Administrator</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8437in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Full Synapse access to SQL pools, Data Explorer pools, Apache Spark pools, and Integration runtimes. Includes create, read, update, and delete access to all published code artifacts. Includes Compute Operator, Linked Data Manager, and Credential User permissions on the workspace system identity credential. Includes assigning Synapse RBAC roles. In addition to Synapse Administrator, Azure Owners can also assign Synapse RBAC roles. Azure permissions are required to create, delete, and manage compute resources.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can read and write artifacts</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can do all actions on Spark activities.</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can view Spark pool logs</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can view saved notebook and pipeline output</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can use the secrets stored by linked services or credentials</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can assign and revoke Synapse RBAC roles at current scope</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.702in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Spark pool</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Integration runtime</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Linked service</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Credential</P> </TD> </TR> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Synapse Apache Spark Administrator</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Full Synapse access to Apache Spark Pools. Create, read, update, and delete access to published Spark job definitions, notebooks and their outputs, and to libraries, linked services, and credentials.&nbsp; Includes read access to all other published code artifacts. Doesn't include permission to use credentials and run pipelines. Doesn't include granting access.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can do all actions on Spark artifacts</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can do all actions on Spark activities</SPAN></P> </TD> <TD style="vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Workspace</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Spark pool</P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse SQL Administrator</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Full Synapse access to serverless SQL pools. Create, read, update, and delete access to published SQL scripts, credentials, and linked services.&nbsp; Includes read access to all other published code artifacts.&nbsp; Doesn't include permission to use credentials and run pipelines. Doesn't include granting access.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can do all actions on SQL scripts</SPAN></P> <P style="margin: 0in; color: black;"><SPAN>Can connect to SQL serverless endpoints with SQL&nbsp;</SPAN><SPAN>db_datareader</SPAN><SPAN>,&nbsp;</SPAN><SPAN>db_datawriter</SPAN><SPAN>,&nbsp;</SPAN><SPAN>connect</SPAN><SPAN>, and&nbsp;</SPAN><SPAN>grant</SPAN><SPAN>&nbsp;permissions</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace</P> </TD> </TR> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Synapse Contributor</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.827in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Full Synapse access to Apache Spark pools and Integration runtimes. Includes create, read, update, and delete access to all published code artifacts and their outputs, including credentials and linked services.&nbsp; Includes compute operator permissions. Doesn't include permission to use credentials and run pipelines. Doesn't include granting access.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can read and write artifacts</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can view saved notebook and pipeline output</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can do all actions on Spark activities</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Can view Spark pool logs</SPAN></P> </TD> <TD style="vertical-align: top; width: 1.702in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Workspace</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Spark pool</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Integration runtime</P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse Artifact Publisher</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Create, read, update, and delete access to published code artifacts and their outputs. Doesn't include permission to run code or pipelines, or to grant access.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can read published artifacts and publish artifacts</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can view saved notebook, Spark job, and pipeline output</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace</P> </TD> </TR> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Synapse Artifact User</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Read access to published code artifacts and their outputs. Can create new artifacts but can't publish changes or run code without additional permissions.</P> </TD> <TD style="vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Workspace</P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse Compute Operator</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Submit Spark jobs and notebooks and view logs.&nbsp; Includes canceling Spark jobs submitted by any user. Requires additional use credential permissions on the workspace system identity to run pipelines, view pipeline runs and outputs.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can submit and cancel jobs, including jobs submitted by others</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can view Spark pool logs</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.702in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Spark pool</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Integration runtime</P> </TD> </TR> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Synapse Monitoring Operator</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.8437in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Read published code artifacts, including logs and outputs for notebooks and pipeline runs. Includes ability to list and view details of serverless SQL pools, Apache Spark pools, Data Explorer pools, and Integration runtimes. Requires additional permissions to run/cancel pipelines, Spark notebooks, and Spark jobs.</P> </TD> <TD style="vertical-align: top; width: 1.6541in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Workspace</P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse Credential User</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Runtime and configuration-time use of secrets within credentials and linked services in activities like pipeline runs. To run pipelines, this role is required, scoped to the workspace system identity.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Scoped to a credential, permits access to data via a linked service that is protected by the credential (also requires compute use permission)</SPAN></P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Allows execution of pipelines protected by the workspace system identity credential(with additional compute use permission)</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Linked Service</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Credential</P> </TD> </TR> <TR> <TD style="vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;"><SPAN>Synapse Linked Data Manager</SPAN></P> </TD> <TD style="vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Creation and management of managed private endpoints, linked services, and credentials. Can create managed private endpoints that use linked services protected by credentials</P> </TD> <TD style="vertical-align: top; width: 1.6736in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt;">Workspace</P> </TD> </TR> <TR> <TD style="background-color: #f2f2f2; vertical-align: top; width: 2.3652in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Synapse User</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 6.8243in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">List and view details of SQL pools, Apache Spark pools, Integration runtimes, and published linked services and credentials. Doesn't include other published code artifacts.&nbsp; Can create new artifacts but can't run or publish without additional permissions.</P> <P style="margin: 0in; font-family: Calibri; font-size: 11.0pt;">&nbsp;</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;"><SPAN>Can list and read Spark pools, Integration runtimes.</SPAN></P> </TD> <TD style="background-color: #f2f2f2; vertical-align: top; width: 1.8062in; padding: 4pt 4pt 4pt 4pt; border: 1pt solid #A3A3A3;"> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Workspace, Spark pool</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Linked service</P> <P style="margin: 0in; font-family: 'Segoe UI'; font-size: 12.0pt; color: black;">Credential</P> </TD> </TR> </TBODY> </TABLE> </DIV> <P>&nbsp;</P> <P>There are multiple ways that the RBAC roles can be configured. <BR />The easiest and most friendly way is to perform this action using Azure Synapse workspace. (<A href="#" target="_blank" rel="noopener">more</A>)</P> <P>In PowerShell, there are number of PowerShell cmdlets that helps to manage or retrieve the information in Synapse RBAC.&nbsp;</P> <P>&nbsp;</P> <P>Example:</P> <UL> <LI>Get-AzSynapseRoleAssignment - Gets a Synapse Analytics role assignment. (<A href="#" target="_blank" rel="noopener">more</A>)</LI> <LI>New-AzSynapseRoleAssignment - Creates a Synapse Analytics role assignment. (<A href="#" target="_blank" rel="noopener">more</A>)</LI> <LI>Remove-AzSynapseRoleAssignment - Deletes a Synapse Analytics role assignment. (<A href="#" target="_blank" rel="noopener">more</A>)</LI> </UL> <P>One of the key use cases that most customers face difficulties while retrieving or assigning the role-based access control in Azure Synapse Analytics, that they cannot find the correct usernames, group names or the service principal names using the PowerShell cmdlet "Get-AzSynapseRoleAssignment". The PowerShell cmdlet only provides limited information and it's difficult to understand since that contains the object IDs.&nbsp;</P> <P>&nbsp;</P> <P>Example: (Following output is captured from “Get-AzSynapseRoleAssignment”)</P> <DIV id="tinyMceEditorCharithCaldera_5" class="mceNonEditable lia-copypaste-placeholder">&nbsp;</DIV> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_0-1654281769763.png" style="width: 463px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/377502i37825D957A127CDC/image-dimensions/463x278?v=v2" width="463" height="278" role="button" title="CharithCaldera_0-1654281769763.png" alt="CharithCaldera_0-1654281769763.png" /></span></P> <P>&nbsp;</P> <P>In order to retrieve the additional information, users can have following example scripts that provides the username and other relevant information for Azure Synapse RBAC.&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Get all the Synapse RBAC Information:&nbsp;</STRONG></P> <P><BR />Below PowerShell script helps to map the RBAC Object IDs with usernames, groups and the service principals.&nbsp;<BR />The script output provides all the RBAC information in Azure Synapse Analytics Workspace.&nbsp;</P> <P><FONT size="2"><STRONG><EM>Note: This is only an example to retrieve the information and not to use any production code</EM></STRONG></FONT></P> <P><EM><FONT size="2"><STRONG>Script Name:&nbsp;<A href="#" target="_self">GetSynapseRBACInfo.ps1</A></STRONG></FONT></EM></P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="powershell">&lt;# .NOTES Author: Charith Caldera LinkedIn: https://www.linkedin.com/in/charith-caldera-52590a10b/ Email: ccaldera@microsoft.com Last Updated: 2022-06-03 .SYNOPSIS Verify the usernames in Synapse RBAC #UPDATES #&gt; Clear-Host # Map Role Assignment IDs with Roles function GetASynapseRoleDefinition{ param($roleid) try { Get-AzSynapseRoleDefinition -id $roleid -WorkspaceName $workspacename -ErrorAction:SilentlyContinue } catch { Write-Host "Error Occured while getting Synapse RBAC Definitions" } } # Get Synapse RBAC Users function GetSynapseRBACUsers{ try { foreach($profile in $profiles) { $aduser = Get-AzADUser -ObjectId $profile.ObjectId -ErrorAction:SilentlyContinue if($aduser) { $assignments = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.ObjectId} Write-Host "----------------------------------------" Write-Host " " Write-Host "AAD User Identified :" -ForegroundColor Green Write-Host " " Write-Host " User object Found on Object ID" $profile.ObjectId -ForegroundColor Cyan Write-Host " AAD User Principal Name :" $aduser.UserPrincipalName Write-Host " Role Assignment Id :" $assignments.RoleAssignmentId Write-Host " Role Definition Id :" $assignments.RoleDefinitionId $rolename = GetASynapseRoleDefinition $assignments.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $assignments.principalType Write-Host " Scope :" $assignments.Scope Write-Host " " } } } catch { Write-Host "Error Occured while getting Synapse RBAC users" } } # Get Synapse RBAC Groups function GetSynapseRBACGroups{ try { foreach($profile in $profiles) { $adgroup = Get-AzADGroup -ObjectId $profile.ObjectId -ErrorAction:SilentlyContinue if($adgroup) { $assignments = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.ObjectId} Write-Host "----------------------------------------" Write-Host " " Write-Host "AAD Group Identified:" -ForegroundColor Green Write-Host " " Write-Host " Group object Found on Object ID" $profile.ObjectId -ForegroundColor Cyan Write-Host " AAD Group Name :" $adgroup.DisplayName Write-Host " Role Assignment Id :" $assignments.RoleAssignmentId Write-Host " Role Definition Id :" $assignments.RoleDefinitionId $rolename = GetASynapseRoleDefinition $assignments.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $assignments.principalType Write-Host " Scope :" $assignments.Scope Write-Host " " } } } catch { Write-Host "Error Occured while getting Synapse RBAC Groups" } } # Get Synapse RBAC Service Principals function GetSynapseRBACSPs{ try { foreach($profile in $profiles) { $adsps = Get-AzADServicePrincipal -ObjectId $profile.ObjectId -ErrorAction:SilentlyContinue if($adsps) { $assignments = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.ObjectId} Write-Host "----------------------------------------" Write-Host " " Write-Host "Service Principal Identified:" -ForegroundColor Green Write-Host " " Write-Host " Service Principal Found on Object ID" $profile.ObjectId -ForegroundColor Cyan Write-Host " AAD Group Name :" $adsps.DisplayName Write-Host " Role Assignment Id :" $assignments.RoleAssignmentId Write-Host " Role Definition Id :" $assignments.RoleDefinitionId $rolename = GetASynapseRoleDefinition $assignments.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $assignments.principalType Write-Host " Scope :" $assignments.Scope Write-Host " " } } } catch { Write-Host "Error Occured while getting Synapse RBAC Service Principals" } } #################################################### # Main Write-Host "Executing PowerShell Script" -ForegroundColor Green $workspacename = Read-Host "Synapse Workspace Name" $workspacenameob = Get-AzSynapseWorkspace -WorkspaceName $workspacename -ErrorAction:SilentlyContinue if($workspacenameob.Name) { $profiles = Get-AzSynapseRoleAssignment -WorkspaceName $workspacenameob.Name | select ObjectId Write-Host " " if(GetSynapseRBACUsers) { GetSynapseRBACUsers } if(GetSynapseRBACGroups) { GetSynapseRBACGroups } if(GetSynapseRBACSPs) { GetSynapseRBACSPs } } else { Write-Host "Sorry, workspace not found" -ForegroundColor Yellow Write-Host " " } Write-Host "PowerShell Script Completed" -ForegroundColor Green Write-Host " "</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Example: (Following is the output of above script)<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_1-1654282019300.png" style="width: 414px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/377504iF9BC52BC64CCAFAD/image-dimensions/414x455?v=v2" width="414" height="455" role="button" title="CharithCaldera_1-1654282019300.png" alt="CharithCaldera_1-1654282019300.png" /></span></P> <P>&nbsp;</P> <P><STRONG>Get Specific user RBAC information:&nbsp;</STRONG></P> <P>&nbsp;</P> <P>Below PowerShell script helps to map a specific user with Synapse RBAC.&nbsp;<BR />The script provides the information for a specific username, group or a service principal.&nbsp;</P> <P><FONT size="2"><STRONG><EM>Note: This is only an example to retrieve the information and not to use any production code</EM></STRONG></FONT></P> <P><EM><STRONG><FONT size="2">Script Name:&nbsp;<A href="#" target="_self">GetSynapseRBACUser.ps1</A></FONT></STRONG></EM></P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="powershell">&lt;# .NOTES Author: Charith Caldera LinkedIn: https://www.linkedin.com/in/charith-caldera-52590a10b/ Email: ccaldera@microsoft.com Last Updated: 2022-06-03 .SYNOPSIS Find the username in Synapse RBAC #UPDATES #&gt; Clear-Host # Map Role Assignment IDs with Roles function GetASynapseRoleDefinition{ param($roleid) Get-AzSynapseRoleDefinition -id $roleid -WorkspaceName $workspacename } # Get Synapse RBAC Users function GetSynapseRBACUsers{ $profile = Get-AzADUser -UserPrincipalName $username $roleassignment = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.id} if($roleassignment) { Write-Host "AAD Users Identified :" -ForegroundColor Green Write-Host " " Write-Host " Username :" $username Write-Host " Role Assignment Id :" $roleassignment.RoleAssignmentId Write-Host " Role Definition Id :" $roleassignment.RoleDefinitionId $rolename = GetASynapseRoleDefinition $roleassignment.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $roleassignment.principalType Write-Host " Scope :" $roleassignment.Scope Write-Host " " } } # Get Synapse RBAC Groups function GetSynapseRBACGroups{ $profile = Get-AzADGroup -DisplayName $username $roleassignment = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.id} if($roleassignment) { Write-Host "AAD Groups Identified:" -ForegroundColor Green Write-Host " " Write-Host " AAD Group Name :" $username Write-Host " Role Assignment Id :" $roleassignment.RoleAssignmentId Write-Host " Role Definition Id :" $roleassignment.RoleDefinitionId $rolename = GetASynapseRoleDefinition $roleassignment.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $roleassignment.principalType Write-Host " Scope :" $roleassignment.Scope Write-Host " " } } # Get Synapse RBAC Service Principals function GetSynapseRBACSPs{ $profile = Get-AzADServicePrincipal -DisplayName $username $roleassignment = Get-AzSynapseRoleAssignment -WorkspaceName $workspacename |?{$_.ObjectId -eq $profile.id} if($roleassignment) { Write-Host "Service Principals Identified:" -ForegroundColor Green Write-Host " " Write-Host " Service Principal Name :" $username Write-Host " Role Assignment Id :" $roleassignment.RoleAssignmentId Write-Host " Role Definition Id :" $roleassignment.RoleDefinitionId $rolename = GetASynapseRoleDefinition $roleassignment.RoleDefinitionId Write-Host " Synapse Role :" $rolename.Name Write-Host " Principal Type :" $roleassignment.principalType Write-Host " Scope :" $roleassignment.Scope Write-Host " " } } Write-Host "Executing PowerShell Script" -ForegroundColor Green Write-Host " " $workspacename = Read-Host "Synapse Workspace Name" $workspacenameob = Get-AzSynapseWorkspace -WorkspaceName $workspacename -ErrorAction:SilentlyContinue if($workspacenameob.Name) { $username = Read-Host "User Name (name@domain) or Service Principal Display Name" Write-Host " " if(GetSynapseRBACUsers) { GetSynapseRBACUsers } if(GetSynapseRBACGroups) { GetSynapseRBACGroups } if(GetSynapseRBACSPs) { GetSynapseRBACSPs } } else { Write-Host "Sorry, workspace not found" -ForegroundColor Yellow Write-Host " " } Write-Host "PowerShell Script Completed" -ForegroundColor Green Write-Host " "</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Example: (Following output is the output of above script)<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_2-1654282176291.png" style="width: 481px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/377505i2392F1487174E27A/image-dimensions/481x188?v=v2" width="481" height="188" role="button" title="CharithCaldera_2-1654282176291.png" alt="CharithCaldera_2-1654282176291.png" /></span></P> <P>&nbsp;</P> <P><STRONG>How to execute the scripts?&nbsp;</STRONG></P> <P>&nbsp;</P> <P>There are multiple ways you can execute the scripts.</P> <OL> <LI>From local host -&nbsp;<BR /><BR />Copy the commands to a PowerShell script<BR />Rename the PowerShell scripts as "GetSynapseRBACInfo.ps1" &amp; "GetSynapseRBACUser.ps1"<BR />Execute the PowerShell script<BR /><BR /><FONT size="2"><STRONG>Note:&nbsp;Az.Synapse module &amp;&nbsp;Az.Resources modules needs to be installed</STRONG></FONT><BR /><BR /></LI> <LI>From Azure Cloud Shell.&nbsp;<BR /><BR />Upload both scripts to Azure Cloud Shell<BR />Execute the scripts as #1.<BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_3-1654282586610.png" style="width: 547px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/377509iB0BC4FCA475CC823/image-dimensions/547x134?v=v2" width="547" height="134" role="button" title="CharithCaldera_3-1654282586610.png" alt="CharithCaldera_3-1654282586610.png" /></span></LI> </OL> <P>The workspace name, AAD username, AAD user group or Service principal name is required as parameters.&nbsp;<BR />Example: (Following output captured while the script is requesting the necessary information)<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="CharithCaldera_4-1654282788823.png" style="width: 545px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/377510i71A89A4FF6D705FA/image-dimensions/545x92?v=v2" width="545" height="92" role="button" title="CharithCaldera_4-1654282788823.png" alt="CharithCaldera_4-1654282788823.png" /></span></P> <P>&nbsp;</P> <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</P> Mon, 13 Jun 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/retrieve-azure-synapse-role-based-access-control-rbac/ba-p/3466419 CharithCaldera 2022-06-13T15:00:00Z Monitoring Synapse serverless SQL open connections https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/monitoring-synapse-serverless-sql-open-connections/ba-p/3298577 <P><STRONG>Scenario</STRONG>:<STRONG>&nbsp;</STRONG><SPAN>Consider a scenario where you are trying to monitor the connections from other applications to serverless SQL. I hit this need while trying to understand how many connections opened I had coming from an application. Had I hit some kind of limitation on serverless SQL or not. <U>Spoiler</U>: <STRONG>There is no limit</STRONG> for connections on Synapse serverless SQL&nbsp; as you would find with a dedicated SQL pool (formerly SQL DW).</SPAN></P> <P>&nbsp;</P> <P><SPAN>Here it goes how to find how many open connections you have very simply:</SPAN></P> <P><SPAN>1) Open SSMS or Synapse Studio or Azure Data Studio whatever you prefer.</SPAN></P> <P><SPAN>2) Connect to your Synapse&nbsp; serverless SQL endpoint. If you are not connecting from the Synapse Studio, you can find the endpoint on the&nbsp; Azure portal when checking your workspace overview details - Fig 1 Overview (yellow).</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_0-1654075233686.png" style="width: 712px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/376665i9016C03E2D09EB97/image-dimensions/712x153?v=v2" width="712" height="153" role="button" title="Liliam_Leme_0-1654075233686.png" alt="Liliam_Leme_0-1654075233686.png" /></span></P> <P>Fig 1 - Overview</P> <P>&nbsp;</P> <P>3) Now let's check the active sessions and requests. Run the following queries and check the results. This query will expose the sessions and current requests from those sessions in a summarized way.&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">Select DB_NAME(s.database_id) as DBName, COUNT(s.database_id) as NumberOfConnections, nt_user_name as username, login_name as LoginName, program_name as ApplicationName FROM sys.dm_exec_requests req JOIN sys.dm_exec_sessions s ON req.session_id = s.session_id GROUP BY s.database_id, nt_user_name, login_name, program_name</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Another way to monitor is the next query that will summarize the connections per program, login, and database:</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">select DB_NAME(database_id) as DBName, nt_user_name as username, login_name as LoginName, program_name as ApplicationName, host_name, program_name, COUNT(*) AS NB_Connections from sys.dm_exec_sessions GROUP BY DB_NAME(database_id) , nt_user_name , login_name , program_name , host_name, program_name</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>For example, as a result in my own environment I have the following for the second query - Fig 2 Query Results:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Liliam_Leme_1-1654076530883.png" style="width: 584px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/376673i31B3FF9873BDFC4B/image-dimensions/584x375?v=v2" width="584" height="375" role="button" title="Liliam_Leme_1-1654076530883.png" alt="Liliam_Leme_1-1654076530883.png" /></span></P> <P>Fig. 2&nbsp; Query results.</P> <P>&nbsp;</P> <P>As a result, information such as SQL or AAD user, login, application, and hostname will be exposed. As also you may note the results of internal sessions related to the Medatasync service on Serverless. Medatasync does what the name implies sync the metadata with other components. This is internal and is not something a user has control over.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Summary:&nbsp; </STRONG>The&nbsp; dynamic managed views (DMVs) above provide you a way to check out the sessions and active requests from those open sessions open. When I used those DMVs to understand the requests I have coming from my application to serverless I could confirm there was no limitation on the open sessions as I mentioned before.</P> <P>Following are some examples of possible&nbsp; scenarios to use this logic:</P> <UL> <LI>Monitor Active connections</LI> <LI>Monitor users, applications, and hosts connected to the environment.</LI> <LI>Troubleshooting for a suspicious leak of connections can be applied in any scenario.&nbsp;</LI> </UL> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>So for the possible question that you may have up to this point... But...</STRONG><STRONG>Would that mean is there a hard limit on connections inside serverless?</STRONG></P> <P>&nbsp;</P> <P>The short answer is <STRONG>No</STRONG>. What must be considered for&nbsp;serverless is not a limited number of connections but the number of active running queries.</P> <P><U>Considerations about the active running queries</U>:</P> <UL> <LI class="lia-indent-padding-left-30px">The number of active sessions and requests would depend on the query complexity and amount of data scanned.&nbsp;&nbsp;</LI> <LI class="lia-indent-padding-left-30px">I mean...As with any SQL, a serverless SQL pool could handle many different sessions that are executing lightweight queries or complex heavy queries consuming most of the resources while the other queries wait.</LI> </UL> <P><U>And this is documented as follows:</U></P> <P class="lia-indent-padding-left-30px"><EM><SPAN>Check if there is a concurrent workload running on the serverless pool because the other queries might take the resources. In that case you might split the workload on multiple workspaces."</SPAN>&nbsp;</EM></P> <P class="lia-indent-padding-left-30px"><A href="#" target="_self">Self-help for serverless SQL pool</A></P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P>&nbsp;</P> <P>Note: Thanks to Sergio Fonseca, Silas Mendes, and Ali Saleh on the CSS teamwork side.</P> <P>&nbsp;</P> <P>Liliam C Leme</P> <P>UK Engineer</P> Wed, 08 Jun 2022 22:43:44 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/monitoring-synapse-serverless-sql-open-connections/ba-p/3298577 Liliam_Leme 2022-06-08T22:43:44Z Recapping Build 2022 https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/recapping-build-2022/ba-p/3457460 <P><SPAN>There was a lot of excitement last week at build. There were a lot of innovations&nbsp;across many Microsoft products including:</SPAN></P> <UL> <LI><SPAN>Synapse Link for SQL</SPAN></LI> <LI><SPAN>SQL server 2022</SPAN></LI> <LI><SPAN>Azure SQL DB</SPAN></LI> <LI><SPAN>Microsoft Purview</SPAN></LI> <LI><SPAN>Power BI</SPAN></LI> </UL> <P>&nbsp;</P> <P><SPAN>As well as the announcement&nbsp;of Microsoft Intelligent Data Platform.</SPAN></P> <P>&nbsp;</P> <P><SPAN> Tune in to hear what we think were some of the most exciting announcements!</SPAN></P> <P>&nbsp;</P> <P>Check out our recap video!</P> <P><IFRAME src="https://www.youtube.com/embed/xNRvccKPRLA" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <P><SPAN>Be sure to subscriber to the YouTube channel to get all of our latest content as well as our Twitter handle&nbsp;<A href="#" target="_self">@Azure_Synapse</A></SPAN></P> <P>&nbsp;</P> <P><SPAN>Announcement Links:</SPAN></P> <P>&nbsp;</P> <P><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/announcing-the-public-preview-of-azure-synapse-link-for-sql/ba-p/3372986%20" target="_self"><SPAN> Azure Synapse Link for SQL</SPAN></A></P> <P><SPAN><A href="#" target="_self">SQL Server 2022 public preview</A></SPAN></P> <P><SPAN><A href="#" target="_self">Dev environment for Azure SQL DB</A></SPAN></P> <P><SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/microsoft-purview-blog/microsoft-purview-data-policy-for-sql-devops-access-provisioning/ba-p/3403174" target="_self">Microsoft Purview Data Policy</A></SPAN></P> <P><SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/microsoft-purview-blog/microsoft-purview-data-estate-insights-will-be-generally/ba-p/3397528" target="_self">Microsoft Purview Data Estate</A></SPAN></P> <P><SPAN><A href="#" target="_self">Cosmos DB updates</A></SPAN></P> <P><SPAN><A href="#" target="_self">Power BI Datamarts</A></SPAN></P> <P>&nbsp;</P> <P><SPAN><A href="#" target="_self">Microsoft Build 2022 Book of News</A>:&nbsp;</SPAN></P> <UL> <LI><A href="#" target="_self"><SPAN>1.3.1: Intelligent Data Platform (IDP) </SPAN></A></LI> <LI><A href="#" target="_self"><SPAN>1.3.2: SQL Server 2022 went Public Preview </SPAN></A></LI> <LI><A href="#" target="_self"><SPAN>1.3.3: Cosmos DB 1.3.4: Dev environment for Azure SQL DB</SPAN></A></LI> <LI><A href="#" target="_self"><SPAN>1.3.5: Azure Synapse Link for SQL &amp; Microsoft Graph Data Connect</SPAN></A></LI> <LI><A href="#" target="_self"><SPAN>1.3.7: Azure Purview Data Estate</SPAN></A></LI> </UL> Thu, 02 Jun 2022 17:50:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/recapping-build-2022/ba-p/3457460 ryanmajidi 2022-06-02T17:50:00Z Azure Synapse Analytics May Update 2022 https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-may-update-2022/ba-p/3430970 <DIV id="generatedtoc"> <H2>Azure Synapse Analytics May 2022 Update</H2> </DIV> <P><SPAN data-contrast="auto">Welcome to the May 2022 update for Azure Synapse Analytics! This month, you’ll find sections on the new Azure Synapse Influencer program,&nbsp;</SPAN><SPAN data-contrast="none">Synapse Data Explorer live query in Excel, a&nbsp;</SPAN><SPAN data-contrast="none">Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse, how to e</SPAN><SPAN data-contrast="none">xport pipeline monitoring as a CSV, and&nbsp;a new&nbsp;Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and PowerApps</SPAN><SPAN data-contrast="auto">.&nbsp;Other new features are in SQL, Spark, Synapse Data Explorer, data integration, and Synapse Link.</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Check out our companion video below!</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/Hzp8oSjEPqA" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <H1>Table of contents</H1> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_0" target="_self" rel="nofollow noopener noreferrer"> General&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_1" target="_self" rel="nofollow noopener noreferrer"> Get connected with the Azure Synapse Influencer program&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_2" target="_self" rel="nofollow noopener noreferrer"> SQL&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_3" target="_self" rel="nofollow noopener noreferrer"> Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse Analytics&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_4" target="_self" rel="nofollow noopener noreferrer"> Specify character column lengths... Not anymore!&nbsp;&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_5" target="_self" rel="nofollow noopener noreferrer"> Apache Spark for Synapse&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_6" target="_self" rel="nofollow noopener noreferrer"> Azure Synapse Dedicated SQL Pool Connector for Apache Spark Now Available in Python&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_7" target="_self" rel="nofollow noopener noreferrer"> Manage Azure Synapse Apache Spark configuration&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_8" target="_self" rel="nofollow noopener noreferrer"> Synapse Data Explorer&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_9" target="_self" rel="nofollow noopener noreferrer"> Synapse Data Explorer live query in Excel&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_11" target="_self" rel="nofollow noopener noreferrer"> Use Managed Identities for External SQL Server Tables&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_12" target="_self" rel="nofollow noopener noreferrer"> New KQL Learn module (2 out of 3) is live!&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_13" target="_self" rel="nofollow noopener noreferrer"> Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps [Generally Available]&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_15" target="_self" rel="nofollow noopener noreferrer"> Dynamic events routing from event hub to multiple databases&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_16" target="_self" rel="nofollow noopener noreferrer"> Configure a database using a KQL inline script as part of JSON ARM deployment template&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_17" target="_self" rel="nofollow noopener noreferrer"> Data Integration&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_18" target="_self" rel="nofollow noopener noreferrer"> Export pipeline monitoring as a CSV&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_19" target="_self" rel="nofollow noopener noreferrer"> Incremental data loading made easy for Synapse and Azure Database for PostgreSQL and MySQL&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_20" target="_self" rel="nofollow noopener noreferrer"> User-Defined Functions for Mapping Data Flows [Public Preview]&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_21" target="_self" rel="nofollow noopener noreferrer"> Assert Error Handling&nbsp; </A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_22" target="_self" rel="nofollow noopener noreferrer"> Mapping data flows projection editing&nbsp; </A></LI> </UL> </LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_23" target="_self" rel="nofollow noopener noreferrer"> Synapse Link&nbsp; </A> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#TOCREF_24" target="_self" rel="nofollow noopener noreferrer"> Azure Synapse Link for SQL [Public Preview]&nbsp;</A></LI> </UL> </LI> </UL> <P>&nbsp;</P> <H2 id="TOCREF_0" aria-level="2"><STRONG><SPAN data-contrast="auto">General</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_1" aria-level="3"><SPAN data-contrast="none">Get connected with the Azure Synapse Influencer program</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><A href="#" target="_blank" rel="noopener"><STRONG><SPAN data-contrast="none">Join a community of Azure Synapse Influencers</SPAN></STRONG></A><STRONG><SPAN data-contrast="none"> who are helping each other achieve more with cloud analytics! </SPAN></STRONG><SPAN data-contrast="none">The Azure Synapse Influencer program</SPAN> <SPAN data-contrast="none">recognizes Azure Synapse Analytics users and advocates who actively support the community by sharing Synapse-related content, announcements, and product news via social media.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">Azure Synapse Influencers range in expertise and engagement level, but they all share one common goal—to </SPAN><STRONG><SPAN data-contrast="none">help others get connected and expand their knowledge of Azure Synapse</SPAN></STRONG><SPAN data-contrast="none">. In addition to recognizing users who regularly create and post content, this program encourages new users to get involved in community discussions.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">If you’re passionate about Azure Synapse Analytics and helping your peers learn and do more with cloud analytics, this program is for you. </SPAN><STRONG><SPAN data-contrast="none">The Azure Synapse Influencer Program brings together like-minded data professionals </SPAN></STRONG><SPAN data-contrast="none">through social media to create a community of influencers who help each other solve challenges, get connected with a global network of users, and achieve more through Azure Synapse. Azure Synapse Influencer members will also have opportunities to collaborate directly with the Azure Synapse engineering team to create new content and highlight their expertise on specific topics.&nbsp;&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">Azure Synapse Influencers and advocates of </SPAN><STRONG><SPAN data-contrast="none">all levels are welcome to join the program and becoming a member is easy! </SPAN></STRONG><SPAN data-contrast="none">If you’re interested in helping to build an active community by engaging with and producing Synapse-related content, we encourage you to </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">sign up for the Azure Synapse Influencer program</SPAN></A><SPAN data-contrast="none"> today!</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> <H2 id="TOCREF_2" aria-level="2"><STRONG><SPAN data-contrast="auto">SQL</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_3" aria-level="3"><SPAN data-contrast="none">Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse Analytics</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">With the </SPAN><SPAN data-contrast="auto">benefits</SPAN><SPAN data-contrast="none"> that </SPAN><SPAN data-contrast="auto">cloud migration</SPAN><SPAN data-contrast="none"> offers, we hear that you often look for steps, processes, or guidelines to follow for quick and easy migrations from existing data warehouse environments. We just released a set of </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse migration guides</SPAN></A><SPAN data-contrast="none"> to make your transition to </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">dedicated SQL Pools in Azure Synapse Analytics</SPAN></A> <SPAN data-contrast="none">easier.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">The </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse Migration Guide</SPAN></A><SPAN data-contrast="none"> provides tips and best practices to make migrating workloads to dedicated SQL pools easier. It also provides guidance on the optimal approaches for migrations around Security.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">The Synapse migration guide for </SPAN><SPAN data-contrast="auto">Teradata</SPAN><SPAN data-contrast="none"> and </SPAN><SPAN data-contrast="auto">IBM Netezza</SPAN><SPAN data-contrast="none"> contains the following:&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI><STRONG><SPAN data-contrast="none">Design and Performance</SPAN></STRONG><SPAN data-contrast="none"> – Learn all about the design differences between on-premises data sources and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse Analytics</SPAN></A><SPAN data-contrast="none"> and performance tuning approaches to adopt.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><STRONG><SPAN data-contrast="none">ETL and load Migration considerations</SPAN></STRONG><SPAN data-contrast="none"> – Read about the initial decisions to be considered for ETL and Data Migrations and best migration approaches to minimize your risk and impact.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><STRONG><SPAN data-contrast="none">Security, access, and Operations for Migrations</SPAN></STRONG><SPAN data-contrast="none"> – Learn how to connect and migrate to </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Dedicated SQL Pools in Azure Synapse Analytics</SPAN></A><SPAN data-contrast="none">. Also, find workload management and automation considerations for housekeeping tasks here.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><STRONG><SPAN data-contrast="none">Visualization and Reporting -</SPAN></STRONG><SPAN data-contrast="none"> This section provides guidance on migrating dashboards and reports.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="5" data-aria-level="1"><STRONG><SPAN data-contrast="none">Minimize SQL Issues for Migrations</SPAN></STRONG><SPAN data-contrast="none"> – Read recommendations to simplify the task of migrating code (Stored procedures, DDL, DML, etc.).&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="6" data-aria-level="1"><STRONG><SPAN data-contrast="none">Microsoft and 3rd Party Tools</SPAN></STRONG><SPAN data-contrast="none"> – Learn which Microsoft and Microsoft Partner tools you can use to migrate your data.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="7" data-aria-level="1"><STRONG><SPAN data-contrast="none">Implementing Modern Data Warehouse</SPAN></STRONG><SPAN data-contrast="none"> – Learn how to leverage other technologies within the Microsoft analytical eco-system to modernize your Data Warehouses.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-contrast="none">Coming soon are guides for Oracle and SQL Server.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">Check out the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse Migration guides</SPAN></A><SPAN data-contrast="none"> and embark on your journey into Azure Synapse Analytics.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_4" aria-level="3"><SPAN data-contrast="none">Specify character column lengths... Not anymore!&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Serverless SQL pools let you query files in the data lake without knowing the schema upfront. The best practice was to specify the lengths of character columns to get optimal performance. Not anymore!&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Previously, you had to explicitly define the schema to get optimal query performance. In this case, the column countries_and_territories is defined as varchar(50):&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="applescript">SELECT * FROM OPENROWSET( BULK 'https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/ecdc_cases/latest/ecdc_cases.csv', FORMAT = 'CSV', PARSER_VERSION = '2.0', HEADER_ROW = TRUE) WITH ( countries_and_territories VARCHAR(50) ) AS [r] </LI-CODE> <P>&nbsp;</P> <P><SPAN data-contrast="auto">With this new feature, you can get optimal query performance without having to define the schema. The serverless SQL pool will calculate the average column length for each inferred character column or character column defined as larger than 100 bytes. The schema will stay the same, while the serverless SQL pool will use the calculated average column lengths internally. It will also automatically calculate the cardinality estimation in case there was no previously created statistic. This feature works on sampled data before a user query is executed. Both average column lengths and cardinality estimations are stored and reused for other queries targeting the same source. In this case, the column countries_and_territories is inferred as varchar(8000) while calculated average column length will be used for resource estimations:&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="applescript">SELECT countries_and_territories FROM OPENROWSET( BULK 'https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/ecdc_cases/latest/ecdc_cases.csv', FORMAT = 'CSV', PARSER_VERSION = '2.0', HEADER_ROW = TRUE) AS [r] </LI-CODE> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Character column lengths are used for query resource estimations as well as cardinality estimations when there are no previously created statistics. Both have a significant impact on query performance. With this recent improvement, you can stop worrying about schema and focus on delivering value to your customers.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The best part? It works for both OPENROWSET and external tables out of the box, as it will be triggered whenever you omit schema or specify character column length larger than 100 bytes.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H2 id="TOCREF_5" aria-level="2"><STRONG><SPAN data-contrast="auto">Apache Spark for Synapse</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_6" aria-level="3"><SPAN data-contrast="none">Azure Synapse Dedicated SQL Pool Connector for Apache Spark Now Available in Python</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Previously, the Azure Synapse Dedicated SQL Pool connector was only available using Scala.&nbsp; Now, it can be used with Python on Spark 3.&nbsp; The only difference between the Scala and Python implementations is the optional Scala callback handle, which allows you to receive post-write metrics.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The following are now supported in Python on Spark 3:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="10" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="auto">Read using AAD Authentication or Basic Authentication</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="10" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="auto">Write to Internal Table using AAD Authentication or Basic Authentication</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="10" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><SPAN data-contrast="auto">Write to External Table using AAD Authentication or Basic Authentication&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> </UL> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Below is an example of Read using AAD Authentication:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="applescript"># Add required imports import com.microsoft.spark.sqlanalytics from com.microsoft.spark.sqlanalytics.Constants import Constants from pyspark.sql.functions import col # Read from existing internal table dfToReadFromTable = (spark.read # If `Constants.SERVER` is not provided, the `&lt;database_name&gt;` from the three-part table name argument # to `synapsesql` method is used to infer the Synapse Dedicated SQL End Point. .option(Constants.SERVER, "&lt;sql-server-name&gt;.sql.azuresynapse.net") # Defaults to storage path defined in the runtime configurations .option(Constants.TEMP_FOLDER, "abfss://&lt;container_name&gt;@&lt;storage_account_name&gt;.dfs.core.windows.net/&lt;some_base_path_for_temporary_staging_folders&gt;") # Three-part table name from where data will be read. .synapsesql("&lt;database_name&gt;.&lt;schema_name&gt;.&lt;table_name&gt;") # Column-pruning i.e., query select column values. .select("&lt;some_column_1&gt;", "&lt;some_column_5&gt;", "&lt;some_column_n&gt;") # Push-down filter criteria that gets translated to SQL Push-down Predicates. .filter(col("Title").contains("E")) # Fetch a sample of 10 records .limit(10)) # Show contents of the dataframe dfToReadFromTable.show() </LI-CODE> <P>&nbsp;</P> <P><SPAN data-contrast="auto">And Write using AAD Authentication (Internal Table):&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="applescript"># Add required imports import com.microsoft.spark.sqlanalytics from com.microsoft.spark.sqlanalytics.Constants import Constants # Configure and submit the request to write to Synapse Dedicated SQL Pool (df.write # If `Constants.SERVER` is not provided, the `&lt;database_name&gt;` from the three-part table name argument # to `synapsesql` method is used to infer the Synapse Dedicated SQL End Point. .option(Constants.SERVER, "&lt;sql-server-name&gt;.sql.azuresynapse.net") # Like-wise, if `Constants.TEMP_FOLDER` is not provided, the connector will use the runtime staging directory config (see section on Configuration Options for details). .option(Constants.TEMP_FOLDER, "abfss://&lt;container_name&gt;@&lt;storage_account_name&gt;.dfs.core.windows.net/&lt;some_base_path_for_temporary_staging_folders&gt;") # Choose a save mode that is apt for your use case. # Options for save modes are "error" or "errorifexists" (default), "overwrite", "append", "ignore". # refer to https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html#save-modes .mode("overwrite") # Required parameter - Three-part table name to which data will be written .synapsesql("&lt;database_name&gt;.&lt;schema_name&gt;.&lt;table_name&gt;")) </LI-CODE> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more about the connector in Python, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse Dedicated SQL Pool Connector for Apache Spark</SPAN></A>.</P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_7" aria-level="3"><SPAN data-contrast="none">Manage Azure Synapse Apache Spark configuration</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Apache Spark configuration management is always a challenging task because Spark has hundreds of properties. It is also challenging for you to know the optimal value for Spark configurations. With the new Spark configuration management feature, you can create a standalone Spark configuration artifact with auto-suggestions and built-in validation rules. The Spark configuration artifact allows you to share your Spark configuration within and across Azure Synapse workspaces. You can also easily associate your Spark configuration with a Spark pool, a Notebook, and a Spark job definition for reuse and minimize the need to copy the Spark configuration in multiple places.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_0-1653594596051.png" style="width: 900px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375558i90C95FE50F15201C/image-dimensions/900x416?v=v2" width="900" height="416" role="button" title="kschaaf_0-1653594596051.png" alt="kschaaf_0-1653594596051.png" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about the new Spark configuration management feature, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Manage Apache Spark configuration</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P aria-level="2"><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P aria-level="2">&nbsp;</P> <H2 id="TOCREF_8" aria-level="2"><STRONG><SPAN data-contrast="auto">Synapse Data Explorer</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_9"><SPAN data-contrast="none">Synapse Data Explorer live query in Excel</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">Using the new Data Explorer web experience </SPAN><I><SPAN data-contrast="none">Open in Excel</SPAN></I><SPAN data-contrast="none"> feature, you can now provide access to live results of your query by sharing the connected Excel Workbook with colleagues and team members.  You can open the live query in an Excel Workbook and refresh it directly from Excel to get the most up to date query results.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><STRONG><SPAN data-contrast="none">Using </SPAN></STRONG><STRONG><I><SPAN data-contrast="none">Open in Excel</SPAN></I></STRONG><STRONG><SPAN data-contrast="none"> option In Synapse Data Explorer</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To create an Excel Workbook connected to Synapse Data Explorer, start by </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">running a query in the Web experience</SPAN></A><SPAN data-contrast="none">. Once you have a result set you are happy with, click the 'Open in Excel' button, located under the 'Share' menu:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_1-1653594596052.png" style="width: 299px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375559i460E0ECEB78A0DEB/image-dimensions/299x293?v=v2" width="299" height="293" role="button" title="kschaaf_1-1653594596052.png" alt="kschaaf_1-1653594596052.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">The WebUI will create and download an Excel workbook that is connected to Data Explorer. Open the downloaded file and continue your work in Excel.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><STRONG><SPAN data-contrast="none">Excel integration</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Once you’ve opened the exported file in Excel, the query results will be displayed. You may need to enter credentials to access the data.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">The Excel workbook that was just created is connected to Synapse Data Explorer and is refreshable. Simply refresh your Excel workbook with new data from Data Explorer and all related Excel artifacts, like charts and pivot tables, will be updated with the latest data.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">This new integration between Synapse Data Explorer and Excel is yet another way to help you be more productive using Azure data and the Office suite.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">To learn more about Excel live query, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-explorer-blog/open-live-kusto-query-in-excel/ba-p/3198500" target="_blank" rel="noopener"><SPAN data-contrast="none">Open live query in Excel</SPAN></A>.<SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_10" aria-level="3">&nbsp;</H3> <H3 id="TOCREF_11" aria-level="3"><SPAN data-contrast="none">Use Managed Identities for External SQL Server Tables</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">One of the key benefits of Azure Synapse is the ability to bring together data integration, enterprise data warehousing, and big data analytics. With Managed Identity support, Synapse Data Explorer table definition is now simpler and more secure. You can now use managed identities instead of entering in your credentials.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">An external SQL table is a schema entity that references data stored outside the Synapse Data Explorer database. Using the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Create and alter SQL Server external tables</SPAN> <SPAN data-contrast="none">command</SPAN></A><SPAN data-contrast="none">, External SQL tables can easily be added to the Synapse Data Explorer database schema.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">The following option was added to the SqlServerConnectionString in the Create/Alter Table command:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="22" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">Authentication="Active Directory Managed Identity"</SPAN><SPAN data-contrast="none"> for a </SPAN><I><SPAN data-contrast="none">system-assigned managed identity</SPAN></I><SPAN data-contrast="none"> or&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="16" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="5" data-aria-level="1"><SPAN data-contrast="none">Authentication="Active Directory Managed Identity";User Id={object_id}</SPAN><SPAN data-contrast="none"> for a </SPAN><I><SPAN data-contrast="none">user-assigned managed identity</SPAN></I><SPAN data-contrast="none"> with its object ID.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To learn more about managed identities, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Managed identities overview</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about external tables, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Create and alter SQL Server external tables</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H3 id="TOCREF_12"><SPAN data-contrast="none">New KQL Learn module (2 out of 3) is live!</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">The power of Kusto Query Language (KQL) is its simplicity to query structured, semi-structured, and unstructured data together. To make it easier for you to learn KQL, we are releasing Learn modules. Previously, we released </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Write your first query with Kusto Query Language</SPAN></A><SPAN data-contrast="none">. New this month is </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Gain insights from your data by using Kusto Query Language</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">KQL is the query language used to query Synapse Data Explorer big data. KQL has a fast-growing user community, with hundreds of thousands of developers, data engineers, data analysts, and students.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">Check out the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">newest KQL Learn Model</SPAN></A><SPAN data-contrast="none"> and see for yourself how easy it is to become a KQL master.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">To learn more about KQL, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Kusto Query Language (KQL) overview</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H3 id="TOCREF_13" aria-level="3"><SPAN data-contrast="none">Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps [Generally Available]</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">The Azure Data Explorer connector for Power Automate enables you to orchestrate and schedule flows, send notifications, and alerts, as part of a scheduled or triggered task.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">In Power Automate, you can:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="15" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:510,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="6" data-aria-level="1"><SPAN data-contrast="none">Send notifications and alerts based on query results, such as when thresholds exceed certain limits.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="15" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:510,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="7" data-aria-level="1"><SPAN data-contrast="none">Send regular, such as daily or weekly, reports containing tables and charts.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="15" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:510,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="8" data-aria-level="1"><SPAN data-contrast="none">Schedule regular jobs using control commands on clusters. For example, copy data from one table to another using the .set-or-append command.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="15" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:510,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="9" data-aria-level="1"><SPAN data-contrast="none">Export and import data between Azure Data Explorer and other databases</SPAN><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To create a new flow using the Azure Data Explorer connector:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <OL> <LI data-leveltext="%1." data-font="" data-listid="21" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Sign-in to Power Automate</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="" data-listid="21" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Create a New scheduled Cloud Flow</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="" data-listid="21" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Add a Data Explorer operation by selecting one of the supported Azure Data Explorer actions:&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_2-1653594596054.png" style="width: 435px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375560i93A41BF763F89EBA/image-dimensions/435x556?v=v2" width="435" height="556" role="button" title="kschaaf_2-1653594596054.png" alt="kschaaf_2-1653594596054.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">The supported Azure Data Explorer actions are:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="20" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="10" data-aria-level="1"><SPAN data-contrast="none">Run KQL Query: </SPAN><SPAN data-contrast="none">This action sends a query to the specified cluster. </SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="20" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="11" data-aria-level="1"><SPAN data-contrast="none">Run KQL query and render a chart: </SPAN><SPAN data-contrast="none">Use this action to visualize a KQL query result as a table or chart. </SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="20" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="12" data-aria-level="1"><SPAN data-contrast="none">Run async control command: </SPAN><SPAN data-contrast="none">This action runs control command in async mode and returns its ID, state, and status on completion.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="20" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="13" data-aria-level="1"><SPAN data-contrast="none">Run control command and render a chart: </SPAN><SPAN data-contrast="none">Use this action to run a control command and get the result as a chart of your choice.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="20" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="14" data-aria-level="1"><SPAN data-contrast="none">Run show control command:&nbsp; This action runs the show control command and returns the result.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-contrast="none">In addition, you can include a step in any flow to send reports by email, to any email address.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">To learn more about the Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Data Explorer connector for Microsoft Power Automate</SPAN></A> <SPAN data-contrast="auto">and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Usage examples for Azure Data Explorer connector to Power Automate</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H2 id="TOCREF_14" aria-level="2"><STRONG><SPAN data-contrast="auto">The following features are Standalone Azure Data Explorer features. Linked services can be used to connect the Azure Data Explorer (ADX) databases to Synapse.</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_15" aria-level="3"><SPAN data-contrast="none">Dynamic events routing from event hub to multiple databases</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">Routing events from Event Hub/IOT Hub/Event Grid is an activity commonly performed by Azure Data Explorer (ADX) users. Previously, you could route events only to a single database per defined connection. If you wanted to route the events to multiple databases, you needed to create multiple ADX cluster connections.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To simplify the experience, we now support routing events data to multiple databases hosted in a single ADX cluster.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_3-1653594596056.png" style="width: 534px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375562iDE957176EF2F04BA/image-dimensions/534x316?v=v2" width="534" height="316" role="button" title="kschaaf_3-1653594596056.png" alt="kschaaf_3-1653594596056.png" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">Routing data to an alternate database is off by default. To send the data to a different database, you must first set the connection as a multi-database connection. You can do this in the Azure portal </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure portal</SPAN></A><SPAN data-contrast="none">, </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">C#</SPAN></A><SPAN data-contrast="none">, </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Python</SPAN></A><SPAN data-contrast="none">, or an </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">ARM template</SPAN></A><SPAN data-contrast="none">. </SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="none">For example, the configuration option in the Portal is:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_4-1653594596057.png" style="width: 626px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375561iBC1C891432CDA1B8/image-dimensions/626x101?v=v2" width="626" height="101" role="button" title="kschaaf_4-1653594596057.png" alt="kschaaf_4-1653594596057.png" /></span></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:true,&quot;134233118&quot;:true,&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">To learn more about dynamic routing, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Ingest from event hub</SPAN></A><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_16" aria-level="3"><SPAN data-contrast="none">Configure a database using a KQL inline script as part of JSON ARM deployment template</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="none">Previously, Azure Data Explorer supported running a Kusto Query Language (KQL) script to configure your database during Azure Resource Management (ARM) template deployment. Now, this can be done using an inline script. The script is provided inline as a parameter to a JSON ARM template.</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">A script is a list of one or more </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">control commands</SPAN></A><SPAN data-contrast="none">, each separated by one line break, and is created as a resource that will be accessed with the ARM template. The script can only run control commands that start with:</SPAN><SPAN data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="o" data-font="Courier New" data-listid="12" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:360,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">.create</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="o" data-font="Courier New" data-listid="12" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:360,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="none">.create-or-alter</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="o" data-font="Courier New" data-listid="12" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:360,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><SPAN data-contrast="none">.create-merge</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="o" data-font="Courier New" data-listid="12" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:360,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="4" data-aria-level="1"><SPAN data-contrast="none">.alter</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> <LI data-leveltext="o" data-font="Courier New" data-listid="12" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:360,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="5" data-aria-level="1"><SPAN data-contrast="none">.alter-merge</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559685&quot;:1290,&quot;335559739&quot;:0,&quot;335559740&quot;:240}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="none">The following is an example of how the inline parameter is defined in the JSON ARM template:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="applescript"> "kqlScript": { "defaultValue": ".create-merge table MyTable (Level:string, Timestamp:datetime, UserId:string, TraceId:string, Message:string, ProcessId:int32)\n\n.create-merge table MyTable2 (Level:string, Timestamp:datetime, UserId:string, TraceId:string, Message:string, ProcessId:int32)", "type": "String" }, </LI-CODE> <P>&nbsp;</P> <P><SPAN data-contrast="none">To learn more about using a KQL inline script, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Configure a database using a Kusto Query Language script</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> <H2 id="TOCREF_17" aria-level="2"><STRONG><SPAN data-contrast="auto">Data Integration</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <H3 id="TOCREF_18" aria-level="3"><SPAN data-contrast="none">Export pipeline monitoring as a CSV</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">The ability to export pipeline monitoring to CSV has been added after receiving many community requests for the feature. Simply filter the Pipeline runs screen to the data you want and click ‘Export to CSV’.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_5-1653594596059.png" style="width: 784px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375563i5800AFB2AC262F3B/image-dimensions/784x384?v=v2" width="784" height="384" role="button" title="kschaaf_5-1653594596059.png" alt="kschaaf_5-1653594596059.png" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about exporting pipeline monitoring and other monitoring improvements, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-factory-blog/adf-monitoring-improvements/ba-p/3295531" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Data Factory monitoring improvements</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_19" aria-level="3"><SPAN data-contrast="none">Incremental data loading made easy for Synapse and Azure Database for PostgreSQL and MySQL</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">In a data integration solution, incrementally loading data after an initial full data load is a widely used scenario. Automatic incremental source data loading is now natively available for Synapse SQL and Azure Database for PostgreSQL and MySQL. With a simple click, users can “enable incremental extract” and only inserted or updated rows will be read by the pipeline.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about incremental data loading, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Incrementally copy data from a source data store to a destination data store</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <H3 id="TOCREF_20" aria-level="3"><SPAN data-contrast="none">User-Defined Functions for Mapping Data Flows [Public Preview]</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">We hear you that you can find yourself doing the same string manipulation, math calculations, or other complex logic several times. Now, with the new user-defined function feature, you can create customized expressions that can be reused across multiple mapping data flows. User-defined functions will be grouped in libraries to help developers group common sets of functions.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Once you’ve created a data flow library, you can add in your user-defined functions. You can even add in multiple arguments to make your function more reusable.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_6-1653594596060.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375564i0F5EEE829D13766F/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_6-1653594596060.png" alt="kschaaf_6-1653594596060.png" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about user-defined functions, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-data-factory-blog/introducing-user-defined-functions-preview-for-mapping-data/ba-p/3414628" target="_blank" rel="noopener"><SPAN data-contrast="none">User defined functions in mapping data flows</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_21" aria-level="3"><SPAN data-contrast="none">Assert Error Handling</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">Error handling has now been added to sinks following an assert transformation. Assert transformations enable you to build custom rules for data quality and data validation. You can now choose whether to output the failed rows to the selected sink or to a separate file.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">This example shows which rows failed your assertions:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_7-1653594596061.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375565i7DD7D29AB95C666D/image-size/medium?v=v2&amp;px=400" role="button" title="kschaaf_7-1653594596061.png" alt="kschaaf_7-1653594596061.png" /></span></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more about error handling, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Assert data transformation in mapping data flow</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H3 id="TOCREF_22" aria-level="3"><SPAN data-contrast="none">Mapping data flows projection editing</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">New UI updates have been made to source projection editing in mapping data flows. You can now update source projection column names and column types with the click of a button.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kschaaf_8-1653594596062.png" style="width: 569px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/375566i470D5C6EBE861390/image-dimensions/569x276?v=v2" width="569" height="276" role="button" title="kschaaf_8-1653594596062.png" alt="kschaaf_8-1653594596062.png" /></span></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more about source projection editing, read </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Source transformation in mapping data flow</SPAN></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P aria-level="2"><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H2 id="TOCREF_23" aria-level="2"><STRONG><SPAN data-contrast="auto">Synapse Link</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P><LI-WRAPPER></LI-WRAPPER></P> <H3 id="TOCREF_24" aria-level="3"><SPAN data-contrast="none">Azure Synapse Link for SQL [Public Preview]</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H3> <P><SPAN data-contrast="auto">At Microsoft Build 2022, we announced the Public Preview availability of Azure Synapse Link for SQL, for both </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">SQL Server 2022</SPAN></A><SPAN data-contrast="auto"> and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure SQL Database</SPAN></A><SPAN data-contrast="auto">. Data-driven, quality insights are critical for companies to stay competitive. The speed to achieve those insights can make all the difference. The costly and time-consuming nature of traditional ETL and ELT pipelines are no longer enough. With this release, you can now take advantage of low- and no-code, near real-time data replication from your SQL-based operational stores into Azure Synapse Analytics. This makes it easier to run BI reporting on operational data in near real-time, with minimal impact on your operational store.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more, read </SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/announcing-the-public-preview-of-azure-synapse-link-for-sql/ba-p/3372986" target="_blank" rel="noopener"><SPAN data-contrast="none">Announcing the Public Preview of Azure Synapse Link for SQL</SPAN></A><SPAN data-contrast="auto"> and watch our YouTube video.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/pgusZy34-Ek" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> Tue, 31 May 2022 15:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/azure-synapse-analytics-may-update-2022/ba-p/3430970 ryanmajidi 2022-05-31T15:00:00Z Data Warehouse Migrations to Dedicated SQL Pools in Azure Synapse Analytics https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/data-warehouse-migrations-to-dedicated-sql-pools-in-azure/ba-p/3424223 <P><SPAN data-contrast="auto">With the plethora of <A href="#" target="_self">benefits</A>&nbsp;</SPAN><SPAN data-contrast="auto">that <A href="#" target="_self">cloud migration</A>&nbsp;</SPAN><SPAN data-contrast="auto">offers, customers look for steps, process, and guidelines that they can follow for quick and easy migrations from their existing data warehouse environments. We just released a set of&nbsp;</SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse migration guides</SPAN></A><SPAN data-contrast="auto">&nbsp;to make your transition easier to <A href="#" target="_self">dedicated SQL Pools in Azure Synapse Analytics</A></SPAN><SPAN data-contrast="auto">. </SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P data-unlink="true"><SPAN data-contrast="auto">The </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse Migration Guide</SPAN></A><SPAN data-contrast="auto">&nbsp;provides the tips and best practices to adapt which helps migrating the workloads to </SPAN>dedicated SQL pools <SPAN data-contrast="auto">easier.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The Synapse migration guide</SPAN><SPAN data-contrast="auto">&nbsp;for </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Teradata</SPAN></A><SPAN data-contrast="auto"> and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">IBM Netezza</SPAN></A><SPAN data-contrast="auto"> contains the following:</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <OL> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Design and Performance</SPAN></STRONG><SPAN data-contrast="auto"> – The key focus of this section includes the design differences between the On-premises source and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse Analytics</SPAN></A><SPAN data-contrast="auto">. The performance tuning approaches that are needed to be adapted earlier on in the migration process is elaborated.&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN><STRONG><SPAN data-contrast="auto">ETL and load Migration considerations</SPAN></STRONG><SPAN data-contrast="auto"> – This section highlights the initial decisions to be considered for ETL and Data Migrations along with the best migration approach that needs to be adapted to minimize the risk and impact on customers.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Security, access, and Operations for Migrations</SPAN></STRONG><SPAN data-contrast="auto"> – This section describes methods to connect and migrate to </SPAN><A href="#" target="_blank" rel="noopener">Dedicated SQL Pools in Azure Synapse Analytics</A><SPAN data-contrast="auto">. Also included are workload management and automation considerations for housekeeping tasks that can be automated and monitored.</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Visualization and Reporting -</SPAN></STRONG><SPAN data-contrast="auto"> This section provides guidance on migrating dashboards and reports.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Minimize SQL Issues for Migrations</SPAN></STRONG><SPAN data-contrast="auto"> – This section has recommendations to simplify the task of migrating the code (Stored procedures, DDL, DML, etc).</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Microsoft and 3</SPAN></STRONG><STRONG><SPAN data-contrast="auto">rd</SPAN></STRONG><STRONG><SPAN data-contrast="auto"> Party Tools</SPAN></STRONG><SPAN data-contrast="auto"> – This section details the different Microsoft tools along with all the Microsoft Partner tools that can be leveraged to easily migrate the existing data warehouse to Azure Synapse Analytics.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="%1." data-font="Calibri" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Implementing Modern Data Warehouse</SPAN></STRONG><SPAN data-contrast="auto"> – This section describes how customers can leverage other technologies within the Microsoft analytical eco-system to modernize their Data Warehouses.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </OL> <P><STRONG><SPAN data-contrast="auto">Conclusion&nbsp;</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse Migration Guide</SPAN></A><SPAN data-contrast="auto"> will not only help to migrate the workloads easily and quickly to dedicated SQL pools in Azure Synapse Analytics</SPAN><SPAN data-contrast="none"><FONT color="#146cac"><U>,</U></FONT>&nbsp;</SPAN><SPAN data-contrast="auto">but also provides guidance on the most optimized approach that can be adapted for migrations centered around Security. Check out the </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Data Warehouse Migration guides</SPAN></A><SPAN data-contrast="auto">&nbsp;and embark on your journey into Azure Synapse Analytics.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> Tue, 24 May 2022 23:05:46 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/data-warehouse-migrations-to-dedicated-sql-pools-in-azure/ba-p/3424223 ajagadish 2022-05-24T23:05:46Z Announcing the Public Preview of Azure Synapse Link for SQL https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/announcing-the-public-preview-of-azure-synapse-link-for-sql/ba-p/3372986 <P><SPAN data-contrast="auto">Data-driven, quality insights are increasingly critical for companies to stay competitive, and the speed to achieve those insights can make all the difference. The costly and time-consuming nature of traditional ETL and ELT pipelines are no longer enough. At Microsoft Build 2022, we’re announcing the public preview availability of Azure Synapse Link for SQL, for both </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">SQL Server 2022</SPAN></A><SPAN data-contrast="auto"> and </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure SQL Database</SPAN></A><SPAN data-contrast="auto">. With this release, you can now take advantage of low- and no-code, near real-time data replication from your SQL-based operational stores into Azure Synapse Analytics. This makes it easier to run BI reporting on operational data in near real-time, with minimal impact on your operational store.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><STRONG><SPAN data-contrast="auto">Sign up today:</SPAN></STRONG><SPAN data-contrast="auto"> Watch Azure Synapse Link for SQL in action in </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">our free webinar</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <HR /><CENTER><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Terence_OShea_0-1653081700140.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373810iE400EA5BEDF30175/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_0-1653081700140.png" alt="Terence_OShea_0-1653081700140.png" /></span></CENTER><CENTER><SPAN><EM>Synapse Link for SQL Server 2022 helps us to seamlessly replicate operational data in near real time to be able to have more powerful analytics. –</EM></SPAN> <SPAN>Javier Villegas, IT Director for DBA and BI Services, Mediterranean Shipping Company</SPAN></CENTER><HR /> <P aria-level="1"><SPAN data-contrast="none">What is Azure Synapse Link for SQL?</SPAN><SPAN data-ccp-props="{&quot;335559738&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Azure Synapse Link for SQL is an automated system for replicating data from your transactional databases (both SQL Server 2022 and Azure SQL Database) into a dedicated SQL pool in Azure Synapse Analytics. The process of setting up a link from your SQL data to Azure Synapse takes just a few clicks and a matter of minutes rather than hours or days for traditional ETL processes. Once configured, your initial data is replicated into the target dedicated SQL pool. After the initial table seeding, changes made to your source data are replicated in near real-time.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">You control the size of the Azure Synapse Analytics dedicated SQL pool as well as the number of cores used to ingest the data, and your source system handles making the data available for ingestion.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P aria-level="1"><SPAN data-contrast="none">Why use it?</SPAN><SPAN data-ccp-props="{&quot;335559738&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Azure Synapse Link for SQL helps streamline data replication into Azure Synapse Analytics for any organization with data in SQL Server 2022 or Azure SQL Database. A few example scenarios where this applies:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Near real-time analytics:</SPAN></STRONG><SPAN data-contrast="auto"> Traditional ETL system run on a periodic schedule. Once the initial data seeding (which is automated) has completed, Azure Synapse Link for SQL replicates your transactional data into Azure Synapse Analytics in near real-time.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Low-code/no-code solutions:</SPAN></STRONG><SPAN data-contrast="auto"> Azure Synapse Link for SQL provides a low- or no-code solution to replicating data. There’s no need to build and manage ETL packages or warehouse schemas—choose which tables to replicate, provide a distribution method and storage architecture (heap/columnstore), and start the connection.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Data consolidation:</SPAN></STRONG><SPAN data-contrast="auto"> Azure Synapse Link for SQL lets you easily combine data from multiple operational databases (both cloud-based and on-premises) into a single cloud-based analytical solution that you can use for workloads including—but not limited to—reporting and machine learning modeling.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Minimal impact on source systems: </SPAN></STRONG><SPAN data-contrast="auto">Batch-based ETL and ELT systems can put extra load on an operational system when they query for data changes. Azure Synapse Link for SQL introduces a new change feed processor that watches for changes and efficiently moves them to a temporary landing zone for processing into the target dedicated SQL pool, thereby minimizing the impact of extracting changes from your source system.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <P aria-level="1"><SPAN data-contrast="none">What makes it work?</SPAN><SPAN data-ccp-props="{&quot;335559738&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_1-1653072235911.png" style="width: 499px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373765iA1A9F22707389142/image-dimensions/499x216?v=v2" width="499" height="216" role="button" title="Terence_OShea_1-1653072235911.png" alt="Terence_OShea_1-1653072235911.png" /></span><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Azure Synapse Link for SQL takes advantage of the latest technology to make data movement simple and efficient:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Change feed: </SPAN></STRONG><SPAN data-contrast="auto">Change feed is a new feature—in both SQL Server 2022 and Azure SQL Database—built to support data synchronization between the source transactional system and Azure Synapse Analytics.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Azure Data Lake Storage Gen2: </SPAN></STRONG><SPAN data-contrast="auto">Azure Synapse Link for SQL takes advantage of Azure Data Lake Storage Gen2 as a landing zone for the data coming from your source systems. This “buffer” helps minimize the impact of the connection on your source database systems.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Azure Synapse Analytics Dedicated SQL Pool:</SPAN></STRONG><SPAN data-contrast="auto"> The ultimate target for the data coming from your source systems is a dedicated SQL pool in Azure Synapse Analytics—which you can size to meet the needs of both your ingestion data volume and your query workloads.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Ingestion Service:</SPAN></STRONG><SPAN data-contrast="auto"> When you enable Azure Synapse Link for SQL, behind the scenes we deploy a cloud-based ingestion service to move data from the landing zone into the target dedicated SQL pool. This is fully managed for you and works in the background to make your data available.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <P aria-level="1"><SPAN data-contrast="none">Getting up and running</SPAN><SPAN data-ccp-props="{&quot;335559738&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To use Azure Synapse Link for SQL, you first need to create the following elements in your Azure Synapse Analytics environment:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Linked Service to Source Database:</SPAN></STRONG><SPAN data-contrast="auto"> This is a standard Azure Synapse Analytics linked service and can be created for either Azure SQL Database or SQL Server 2022.</SPAN><SPAN>&nbsp;<BR /></SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <TABLE data-tablestyle="MsoTableGrid" data-tablelook="1184" aria-rowcount="1"> <TBODY> <TR aria-rowindex="1"> <TD data-celllook="4369"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_2-1653072235914.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373764iBC293273028E7382/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_2-1653072235914.png" alt="Terence_OShea_2-1653072235914.png" /></span> <P>&nbsp;</P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> </TD> <TD data-celllook="4369"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_3-1653072235917.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373766i3AFFB03E3D4C0F9C/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_3-1653072235917.png" alt="Terence_OShea_3-1653072235917.png" /></span> <P>&nbsp;</P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> </TD> <TD data-celllook="4369"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_4-1653072235919.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373767iFB875F96A93BC387/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_4-1653072235919.png" alt="Terence_OShea_4-1653072235919.png" /></span> <P>&nbsp;</P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;335559685&quot;:720}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Azure Synapse Analytics Dedicated SQL Pool:</SPAN></STRONG><SPAN data-contrast="auto"> This will be used as the target for your replicated transactional data.</SPAN><SPAN>&nbsp;<BR /></SPAN> <SPAN>&nbsp;<BR /></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_5-1653072235921.png" style="width: 452px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373768iD2CA45B3B0AF7B6B/image-dimensions/452x201?v=v2" width="452" height="201" role="button" title="Terence_OShea_5-1653072235921.png" alt="Terence_OShea_5-1653072235921.png" /></span></LI> </UL> <P><SPAN data-contrast="auto">If you are using SQL Server 2022 as your source database, you also need to create the following (these are automatically created for Azure SQL Database):</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Azure Data Lake Storage Gen2:</SPAN></STRONG><SPAN data-contrast="auto"> This is the storage account that is used as the landing zone. While this account is managed by you, the files in the landing zone can only be used for Azure Synapse Link for SQL, to allow for changing the retention policy or file format as needed.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Linked Service to Landing Zone:</SPAN></STRONG><SPAN data-contrast="auto"> In the Azure Synapse Analytics workspace, you’ll need to create a linked service pointing to the storage account created above.</SPAN><SPAN>&nbsp;<BR /></SPAN> <SPAN>&nbsp;<BR /></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_6-1653072235922.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373769i97638C5166933852/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_6-1653072235922.png" alt="Terence_OShea_6-1653072235922.png" /></span> <P>&nbsp;</P> <SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><STRONG><SPAN data-contrast="auto">Self-Hosted Integration Runtime:</SPAN></STRONG><SPAN data-contrast="auto"> You’ll also need to configure and install a self-hosted integration runtime that can talk to both the SQL Server 2022 database and the Azure Synapse Analytics environment. This is used to broker commands between the two environments.</SPAN><SPAN>&nbsp;<BR /></SPAN> <SPAN>&nbsp;<BR /></SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_7-1653072235924.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373770i9D0B58EDCEA0E8A1/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_7-1653072235924.png" alt="Terence_OShea_7-1653072235924.png" /></span> <P>&nbsp;</P> </LI> </UL> <P><SPAN data-contrast="auto">Once you have the prerequisites in place, you can create an Azure Synapse Link Connection in Azure Synapse Analytics. When you create the connection, you specify:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">The source database.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">Which tables you’d like to replicate from the source.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">The target dedicated SQL pool.</SPAN><SPAN>&nbsp;<BR /></SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <TABLE data-tablestyle="MsoTableGrid" data-tablelook="1184" aria-rowcount="1"> <TBODY> <TR aria-rowindex="1"> <TD data-celllook="4369"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_8-1653072235926.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373771i6AA88F262915C5B7/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_8-1653072235926.png" alt="Terence_OShea_8-1653072235926.png" /></span> <P>&nbsp;</P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> </TD> <TD data-celllook="4369"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Terence_OShea_9-1653072235928.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373773i2038B9D5F44B359F/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_9-1653072235928.png" alt="Terence_OShea_9-1653072235928.png" /></span> <P>&nbsp;</P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">For each table you set up in the connection, you can specify:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">Table and schema names in the target dedicated SQL pool (these don’t need to be the same as the source table and schema names).</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">Distribution type—you can either use round robin, hash-distributed, or replicated tables.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">Structure type—you can organize your table as either a heap, a clustered columnstore index, or a clustered index.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Detailed instructions—as well as current known limitations—are available in our </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">official documentation</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Once you have completed the initial setup, you can start the connection. At this point, here’s what happens:</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">An initial export of the schema and data from the source tables is executed. This work is actually done by the source database (either Azure SQL DB or SQL Server 2022), and the data is placed into the landing zone.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">The ingestion service picks up the snapshots from the landing zone, creates the target tables in the dedicated SQL pool, and then loads the initial data.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="0" data-aria-level="1"><SPAN data-contrast="auto">Once the initial load is complete, the source database continually publishes changes for each table to the landing zone, where they are picked up by the ingestion service and applied to the target dedicated SQL pool.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">While you cannot make modifications to the configuration of tables in a running link, you can add tables to and remove tables from it.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <HR /><CENTER><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Terence_OShea_0-1653072629415.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/373780i4D4485826ACA7C2A/image-size/medium?v=v2&amp;px=400" role="button" title="Terence_OShea_0-1653072629415.png" alt="Terence_OShea_0-1653072629415.png" /></span></CENTER> <P class="lia-align-center"><I><SPAN data-contrast="none">“With SQL Server 2022 and the arrival of Synapse Link, we can finally achieve near real-time analytics over our most important data, all without the need to build and manage complex packages and pipelines”</SPAN></I><SPAN data-contrast="none"> – Nic Hopper, Data Architect, Atrium Underwriters    </SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <HR /> <P aria-level="1"><SPAN data-contrast="none">Want more information?</SPAN><SPAN data-ccp-props="{&quot;335559738&quot;:240}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more, sign up for </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">our free webinar</SPAN></A><SPAN data-contrast="auto"> where the Azure Synapse Link engineering team will demo the new functionality and answer your questions live.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">You can also review our </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">official documentation</SPAN></A><SPAN data-contrast="auto"> and keep up with the latest news and updates by following </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">@Azure_Synapse</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Plus, don’t miss the launch of our new </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure Synapse Influencers program</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <P aria-level="1"><SPAN data-contrast="none">Learn more about Azure at Microsoft Build:</SPAN></P> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="7" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Read how to </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">build productively, collaborate securely, and scale innovation</SPAN></A><SPAN data-contrast="none">—no matter where in the world with a comprehensive set of Microsoft developer tools and platform.&nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="7" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="none">Read the latest on how </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Azure powers your app innovation and modernization</SPAN></A><SPAN data-contrast="none"> with the choice of control and productivity you need to deploy apps at scale.</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="7" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="none">Read how you can </SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">Innovate faster and achieve greater agility</SPAN></A><SPAN data-contrast="none"> with the Microsoft Intelligent Data Platform and turn your data into decisions. &nbsp;</SPAN><SPAN data-ccp-props="{}">&nbsp;</SPAN></LI> </UL> <P><SPAN data-ccp-props="{}">&nbsp;</SPAN></P> Tue, 24 May 2022 17:46:43 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/announcing-the-public-preview-of-azure-synapse-link-for-sql/ba-p/3372986 chuckheinzelman 2022-05-24T17:46:43Z Introducing the Azure Synapse Influencer Program https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/introducing-the-azure-synapse-influencer-program/ba-p/3389375 <DIV style="text-align: center;"><IFRAME src="https://www.youtube.com/embed/mcD6prCCpNs" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"></IFRAME></DIV> <P><SPAN data-contrast="auto">We’re pleased to announce the </SPAN><A href="#" target="_blank" rel="noopener"><STRONG><SPAN data-contrast="auto">Azure Synapse Influencer program</SPAN></STRONG></A><SPAN data-contrast="auto">. We’re excited to share this program with the community and begin welcoming new members starting today.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The Azure Synapse Analytics community is at the core of everything we do, and this program is an opportunity for us to </SPAN><STRONG><SPAN data-contrast="auto">connect and support community members like you</SPAN></STRONG><SPAN data-contrast="auto">, who are passionate about helping each other achieve more with cloud analytics. Many of you have already been sharing and creating educational content through social media on a regular basis and this program will allow us to formally recognize those efforts and share your knowledge with a wider audience. </SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">We encourage Azure Synapse users, enthusiasts, and advocates of all levels to join the </SPAN><STRONG><SPAN data-contrast="auto">Azure Synapse Influencer program</SPAN></STRONG><SPAN data-contrast="auto">. Azure Synapse Influencers range in expertise and engagement levels, but they all share the common goal of </SPAN><STRONG><SPAN data-contrast="auto">helping others get connected </SPAN></STRONG><SPAN data-contrast="auto">and expanding their knowledge of Azure Synapse. If you’re new to cloud analytics or have just started using Azure Synapse and have discovered content that’s helped you in your analytics journey, we want to share this with others who might be able to benefit from it as well.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><STRONG><SPAN data-contrast="auto">So, how does the program work?</SPAN></STRONG></P> <P><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">If you want to get involved in community discussions around Azure Synapse and help share Azure Synapse-related content via social media, you can now sign up to become an </SPAN><STRONG><SPAN data-contrast="auto">Azure Synapse Influencer</SPAN></STRONG><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Becoming an Azure Synapse Influencer is easy! Simply complete the sign-up form below, select your membership level, follow <A href="#" target="_blank" rel="noopener">@Azure_Synapse</A> on Twitter, and start (or continue) engaging. Share Azure Synapse-related posts and content using the hashtag <A href="#" target="_blank" rel="noopener">#SynapseInfluencers</A>. Once you’ve signed up, you’ll be able to collect your exclusive membership badge and add it to your social media account</SPAN><SPAN data-contrast="none">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener"><STRONG><SPAN data-contrast="none">Sign up now</SPAN></STRONG></A><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The Azure Synapse Influencer program is intended to be accessible and open to everyone in the community. However you want to engage with the program is up to you! As long as you meet the minimum membership requirements, your application is valid, and as an Azure Synapse Influencer you can level up your engagement at any stage of your membership.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><STRONG><SPAN data-contrast="none">Membership levels:&nbsp;</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">There are 3 tiers of Azure Synapse Influencers</SPAN><STRONG><SPAN data-contrast="auto">—</SPAN></STRONG><STRONG><SPAN data-contrast="none">Contender</SPAN></STRONG><STRONG><SPAN data-contrast="auto">, Challenger, and Champion—</SPAN></STRONG><SPAN data-contrast="auto">starting with the most minimal engagement requirements and graduating to more active participation.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <TABLE data-tablestyle="MsoTableGrid" data-tablelook="1696" aria-rowcount="4"> <TBODY> <TR aria-rowindex="1"> <TD data-celllook="65536"> <P><STRONG><SPAN data-contrast="auto">Tier</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> <TD data-celllook="65536"> <P><STRONG><SPAN data-contrast="auto">Membership Requirements</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> </TR> <TR aria-rowindex="2"> <TD data-celllook="0"> <P><STRONG><SPAN data-contrast="auto">Level I - &nbsp;Contender</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> <TD data-celllook="0"> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="auto">Amplify <A href="#" target="_blank" rel="noopener">@Azure_Synapse</A>&nbsp;or <A href="#" target="_blank" rel="noopener">#AzureSynapse</A> social posts (2 likes or shares/month*)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="3" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="auto">Amplify Azure team blogs and/or Azure Synapse channel YouTube videos (1/month)</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> </TD> </TR> <TR aria-rowindex="3"> <TD data-celllook="0"> <P><STRONG><SPAN data-contrast="auto">Level II - &nbsp;Challenger</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> <TD data-celllook="0"> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="auto">All engagement from Level I&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="auto">Post Azure Synapse Analytics-specific content (1 post/month)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> </TD> </TR> <TR aria-rowindex="4"> <TD data-celllook="0"> <P><STRONG><SPAN data-contrast="auto">Level III - &nbsp;Champion</SPAN></STRONG><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> <TD data-celllook="0"> <UL> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="auto">All engagement from Level II&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="1" data-list-defn-props="{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="auto">Post original Azure Synapse content—blog, video, webinar (1 post/month)&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> </TD> </TR> </TBODY> </TABLE> <P><SPAN data-contrast="auto">* Shares must include a Quoted Retweet with an original comment to qualify</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Once you’ve signed up to become a member, you’ll receive your exclusive badge, which you can download and share on your social networks to let others know that you’re a member of the program.</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Azure Synapse Influencer video slides.jpg" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/372626i79959F2609B4CD1D/image-size/large?v=v2&amp;px=999" role="button" title="Azure Synapse Influencer video slides.jpg" alt="Azure Synapse Influencer video slides.jpg" /></span><SPAN>&nbsp;</SPAN></SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">In the coming weeks, </SPAN><STRONG><SPAN data-contrast="auto">we’ll be announcing our first event for Azure Synapse Influencers</SPAN></STRONG><SPAN data-contrast="auto">, where we’ll be answering questions, sharing content, and collecting feedback on the program. We’re also looking forward to creating opportunities for increased collaboration between Azure Synapse Influencers and the Synapse team.</SPAN></P> <P><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">To learn more and join the program, <A href="#" target="_blank" rel="noopener">visit the Azure Synapse Influencer program website</A>, fill out the form, and start sharing your content with the community today!</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN class="">We're excited to announce the launch of our new Azure Synapse Influencer program! Get connected with a global community of data professionals who are passionate about helping each other achieve more with cloud analytics.</SPAN></P> <P>&nbsp;</P> <P><SPAN class="">Azure Synapse Influencers range in expertise and engagement level, but they all share one common goal—<STRONG>to help others get connected and expand their knowledge of Azure Synapse</STRONG>. The Azure Synapse Influencer program recognizes Azure Synapse Analytics community members who actively support others by sharing Synapse-related content, announcements, and product news via social media.</SPAN></P> <P>&nbsp;</P> <P><SPAN class="">Azure Synapse Influencers and advocates of all levels are welcome to join the Azure Synapse Influencer Program. Become an Azure Synapse Influencer today:&nbsp;<A href="#" target="_blank" rel="noopener">http://aka.ms/synapseinfluencers</A></SPAN></P> <P>&nbsp;</P> <P><SPAN class=""><A href="https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/introducing-the-azure-synapse-influencer-program/ba-p/3389375" target="_blank" rel="noopener">Read our recent blog</A> for more information on the program and check out Ryan's new video on the Azure Synapse YouTube channel:</SPAN></P> <P>&nbsp;</P> Mon, 23 May 2022 21:25:05 GMT https://gorovian.000webhostapp.com/?exam=t5/azure-synapse-analytics-blog/introducing-the-azure-synapse-influencer-program/ba-p/3389375 ryanmajidi 2022-05-23T21:25:05Z