How Azure Synapse Analytics can help you respond, adapt, and save

Business disruptions, tactical pivots, and remote work have all emphasized the critical role that analytics plays in all organizations. Uncharted situations demand proven performance insights so that businesses can quickly determine what is and is not working. In recent months, the urgency for business-guiding insights has only been heightened, leading to a need for real-time analytics solutions. Equally important is the need to discover and share these insights in the most cost-effective manner.

Not only has COVID-19 been a challenge to world health but also has created new economic challenges to businesses worldwide. These challenges have resulted in an increased need for tools that quickly deliver insights to business leaders—empowering informed decisions. This is where Microsoft Azure Synapse Analytics can help.

New circumstances demand new solutions

Azure Synapse Analytics is a new type of analytics platform that enables you to accelerate your time-to-insight with a unified experience and—just as important—save on costs while doing so. It is up to 14 times faster and costs 94 percent less than other cloud providers. Let’s dive into how Azure Synapse can help you respond, adapt, and save.

#big data #data science

What is GEEK

Buddha Community

Reform: Form Objects Decoupled From Models In Ruby

Reform

Form objects decoupled from your models.

Reform gives you a form object with validations and nested setup of models. It is completely framework-agnostic and doesn't care about your database.

Although reform can be used in any Ruby framework, it comes with Rails support, works with simple_form and other form gems, allows nesting forms to implement has_one and has_many relationships, can compose a form from multiple objects and gives you coercion.

Full Documentation

Reform is part of the Trailblazer framework. Full documentation is available on the project site.

Reform 2.2

Temporary note: Reform 2.2 does not automatically load Rails files anymore (e.g. ActiveModel::Validations). You need the reform-rails gem, see Installation.

Defining Forms

Forms are defined in separate classes. Often, these classes partially map to a model.

class AlbumForm < Reform::Form
  property :title
  validates :title, presence: true
end

Fields are declared using ::property. Validations work exactly as you know it from Rails or other frameworks. Note that validations no longer go into the model.

The API

Forms have a ridiculously simple API with only a handful of public methods.

1. #initialize always requires a model that the form represents.
2. #validate(params) updates the form's fields with the input data (only the form, not the model) and then runs all validations. The return value is the boolean result of the validations.
3. #errors returns validation messages in a classic ActiveModel style.
4. #sync writes form data back to the model. This will only use setter methods on the model(s).
5. #save (optional) will call #save on the model and nested models. Note that this implies a #sync call.
6. #prepopulate! (optional) will run pre-population hooks to "fill out" your form before rendering.

In addition to the main API, forms expose accessors to the defined properties. This is used for rendering or manual operations.

Setup

In your controller or operation you create a form instance and pass in the models you want to work on.

class AlbumsController
  def new
    @form = AlbumForm.new(Album.new)
  end

This will also work as an editing form with an existing album.

def edit
  @form = AlbumForm.new(Album.find(1))
end

Reform will read property values from the model in setup. In our example, the AlbumForm will call album.title to populate the title field.

Rendering Forms

Your @form is now ready to be rendered, either do it yourself or use something like Rails' #form_for, simple_form or formtastic.

= form_for @form do |f|
  = f.input :title

Nested forms and collections can be easily rendered with fields_for, etc. Note that you no longer pass the model to the form builder, but the Reform instance.

Optionally, you might want to use the #prepopulate! method to pre-populate fields and prepare the form for rendering.

Validation

After form submission, you need to validate the input.

class SongsController
  def create
    @form = SongForm.new(Song.new)

    #=> params: {song: {title: "Rio", length: "366"}}

    if @form.validate(params[:song])

The #validate method first updates the values of the form - the underlying model is still treated as immutuable and remains unchanged. It then runs all validations you provided in the form.

It's the only entry point for updating the form. This is per design, as separating writing and validation doesn't make sense for a form.

This allows rendering the form after validate with the data that has been submitted. However, don't get confused, the model's values are still the old, original values and are only changed after a #save or #sync operation.

Syncing Back

After validation, you have two choices: either call #save and let Reform sort out the rest. Or call #sync, which will write all the properties back to the model. In a nested form, this works recursively, of course.

It's then up to you what to do with the updated models - they're still unsaved.

Saving Forms

The easiest way to save the data is to call #save on the form.

if @form.validate(params[:song])
  @form.save  #=> populates album with incoming data
              #   by calling @form.album.title=.
else
  # handle validation errors.
end

This will sync the data to the model and then call album.save.

Sometimes, you need to do saving manually.

Default values

Reform allows default values to be provided for properties.

class AlbumForm < Reform::Form
  property :price_in_cents, default: 9_95
end

Saving Forms Manually

Calling #save with a block will provide a nested hash of the form's properties and values. This does not call #save on the models and allows you to implement the saving yourself.

The block parameter is a nested hash of the form input.

  @form.save do |hash|
    hash      #=> {title: "Greatest Hits"}
    Album.create(hash)
  end

You can always access the form's model. This is helpful when you were using populators to set up objects when validating.

  @form.save do |hash|
    album = @form.model

    album.update_attributes(hash[:album])
  end

Nesting

Reform provides support for nested objects. Let's say the Album model keeps some associations.

class Album < ActiveRecord::Base
  has_one  :artist
  has_many :songs
end

The implementation details do not really matter here, as long as your album exposes readers and writes like Album#artist and Album#songs, this allows you to define nested forms.

class AlbumForm < Reform::Form
  property :title
  validates :title, presence: true

  property :artist do
    property :full_name
    validates :full_name, presence: true
  end

  collection :songs do
    property :name
  end
end

You can also reuse an existing form from elsewhere using :form.

property :artist, form: ArtistForm

Nested Setup

Reform will wrap defined nested objects in their own forms. This happens automatically when instantiating the form.

album.songs #=> [<Song name:"Run To The Hills">]

form = AlbumForm.new(album)
form.songs[0] #=> <SongForm model: <Song name:"Run To The Hills">>
form.songs[0].name #=> "Run To The Hills"

Nested Rendering

When rendering a nested form you can use the form's readers to access the nested forms.

= text_field :title,         @form.title
= text_field "artist[name]", @form.artist.name

Or use something like #fields_for in a Rails environment.

= form_for @form do |f|
  = f.text_field :title

  = f.fields_for :artist do |a|
    = a.text_field :name

Nested Processing

validate will assign values to the nested forms. sync and save work analogue to the non-nested form, just in a recursive way.

The block form of #save would give you the following data.

@form.save do |nested|
  nested #=> {title:  "Greatest Hits",
         #    artist: {name: "Duran Duran"},
         #    songs: [{title: "Hungry Like The Wolf"},
         #            {title: "Last Chance On The Stairways"}]
         #   }
  end

The manual saving with block is not encouraged. You should rather check the Disposable docs to find out how to implement your manual tweak with the official API.

Populating Forms

Very often, you need to give Reform some information how to create or find nested objects when validateing. This directive is called populator and documented here.

Installation

Add this line to your Gemfile:

gem "reform"

Reform works fine with Rails 3.1-5.0. However, inheritance of validations with ActiveModel::Validations is broken in Rails 3.2 and 4.0.

Since Reform 2.2, you have to add the reform-rails gem to your Gemfile to automatically load ActiveModel/Rails files.

gem "reform-rails"

Since Reform 2.0 you need to specify which validation backend you want to use (unless you're in a Rails environment where ActiveModel will be used).

To use ActiveModel (not recommended because very out-dated).

require "reform/form/active_model/validations"
Reform::Form.class_eval do
  include Reform::Form::ActiveModel::Validations
end

To use dry-validation (recommended).

require "reform/form/dry"
Reform::Form.class_eval do
  feature Reform::Form::Dry
end

Put this in an initializer or on top of your script.

Compositions

Reform allows to map multiple models to one form. The complete documentation is here, however, this is how it works.

class AlbumForm < Reform::Form
  include Composition

  property :id,    on: :album
  property :title, on: :album
  property :songs, on: :cd
  property :cd_id, on: :cd, from: :id
end

When initializing a composition, you have to pass a hash that contains the composees.

AlbumForm.new(album: album, cd: CD.find(1))

More

Reform comes many more optional features, like hash fields, coercion, virtual fields, and so on. Check the full documentation here.

Reform is part of the Trailblazer project. Please buy my book to support the development and learn everything about Reform - there's two chapters dedicated to Reform!

Security And Strong_parameters

By explicitly defining the form layout using ::property there is no more need for protecting from unwanted input. strong_parameter or attr_accessible become obsolete. Reform will simply ignore undefined incoming parameters.

This is not Reform 1.x!

Temporary note: This is the README and API for Reform 2. On the public API, only a few tiny things have changed. Here are the Reform 1.2 docs.

Anyway, please upgrade and report problems and do not simply assume that we will magically find out what needs to get fixed. When in trouble, join us on Gitter.

Full documentation for Reform is available online, or support us and grab the Trailblazer book. There is an Upgrading Guide to help you migrate through versions.

Attributions!!!

Great thanks to Blake Education for giving us the freedom and time to develop this project in 2013 while working on their project.

---

Author: trailblazer
Source code: https://github.com/trailblazer/reform
License:  MIT license

#ruby  #ruby-on-rails

Support for Synapse SQL serverless in Azure Synapse Link for Azure Cosmos DB

Co-authored by Rodrigo Souza, Ramnandan Krishnamurthy, Anitha Adusumilli and Jovan Popovic (Azure Cosmos DB and Azure Synapse Analytics teams)

Azure Synapse Link now supports querying Azure Cosmos DB data using Synapse SQL serverless. This capability, available in public preview, allows you to use familiar analytical T-SQL queries and build powerful near real-time BI dashboards on Azure Cosmos DB data.

As announced at Ignite 2020, you can now also query Azure Cosmos DB API for Mongo DB data using Azure Synapse Link, enabling analytics with Synapse Spark and Synapse SQL serverless.

Support for T-SQL queries and building near real-time BI dashboards

Azure Synapse SQL serverless (previously known as SQL on-demand) is a serverless, distributed data processing service offering built-in query execution fault-tolerance and a consumption-based pricing model. It enables you to analyze your data in Cosmos DB analytical store within seconds, without any performance or RU impact on your transactional workloads.

Using OPENROWSET syntax and automatic schema inference, data and business analysts can use familiar T-SQL query language to quickly explore and reason about the contents in Azure Cosmos DB analytical store. You can query this data in place without the need to copy or load the data into a specialized store.

You can also create SQL views to join data in the analytical stores across multiple Azure Cosmos DB containers, to better organize your data in a semantic layer that will accelerate your data exploration and reporting workloads. BI Professionals can quickly create Power BI reports on top of these SQL views in Direct Query mode.

You can further extend this by building a logical data warehouse to create and analyze unified views of data across Azure Cosmos DB, Azure Data Lake Storage and Azure Blob Storage.

 

#analytics #announcements #api for mongodb #core (sql) api #data architecture #query #azure cosmos db #azure synapse analytics #serverless sql pools #sql on-demand #synapse link #synapse sql serverless

Analyze Azure Cosmos DB data using Azure Synapse Analytics

This article will help you understand how to analyze Azure Cosmos DB data using Azure Synapse Analytics.

Introduction

Azure Cosmos DB is a multi-model NoSQL database that supports hosting various types of data that are transactional in nature. OLTP systems employ transactional databases for hosting operational data. To analyze large volumes of transactional data, relational databases do not scale or perform to the needs of large-scale analytics. Columnar data warehouses are one of the preferred, effective, and proven means of analyzing and aggregating large volumes of data for big data scale analytics. Azure Synapse is the data warehouse offering in the Microsoft Azure technology stack. The challenge with analyzing transactional data in relational databases using columnar warehouses is that one needs to replicate and/or relocate data from operational repositories into analytical repositories. Hybrid transactional analytical processing (HTAP) is a methodology or approach where data hosted in a relational format is auto-organized in a columnar format eliminating the need to replicate and/or relocate the data to a great extent. Azure offers a feature to analyze data hosted in Cosmos DB using Azure Synapse. In this article, we will learn how to implement the same.

Pre-requisites

We are assuming that we are hosting data in the Cosmos DB instance. To simulate this assumption, we would need an Azure Cosmos DB account implemented using the Core (SQL) API, with all the preview features turned on. Once you have an account created, you would be able to see an account listed as shown below.

#azure #sql azure #azure synapse analytics #azure

Integrating Azure Purview with Azure Synapse Analytics

In this article, we will learn how to integrate Azure Purview and Azure Synapse Analytics capabilities to access data catalog assets hosted in Purview from Azure Synapse.

Introduction

Data exists in various formats on various types of repositories on different clouds as well as on-premises. With the growing data landscape, two of the most common capabilities required to manage as well as extract value out of data are data cataloging and data warehousing. Data cataloging or metadata cataloging enables to keep track of the metadata evolution as well acts as a guiding beacon for all data pipelines that move data from source to destination. Data warehousing provides an approach and capabilities to process large volumes of data efficiently when data across the enterprise is collated for deriving insights. The gap between these two capabilities is that if these two capabilities are not integrated, the teams managing these two capabilities would not have any view of each other’s landscape. Typically, the data warehousing capability acts as one of the biggest consumers of data catalogs like many other data capabilities. Azure provides Purview for data cataloging and governance and Azure Synapse Analytics for data warehousing. In this article, we will see how to integrate these two capabilities to access data catalog assets hosted in Azure Purview from Azure Synapse.

Pre-requisite

As we are going to work with Azure Purview as well as Azure Synapse, we need a few things in place before we can start configuring these tools to integrate with each other. It is assumed that one has the required privileges to administer and operate Purview and Azure Synapse services on their Azure account.

First, we need an instance of Purview, which would provide access to the Purview Studio tool. Using this tool, some data repositories should be cataloged so that when we search for data assets cataloged in this tool, we would find some results. A good example would be creating an Azure SQL Database with the sample data that comes built-in and catalog it with Purview. It is assumed that this Azure Purview setup is already in place and data assets are already cataloged.

Next, we need an instance of Azure Synapse Workspace created, which would provide access to the Synapse Studio tool. This is the primary administrative console that facilitates operating the Synapse pool. Once this setup is in place, it would look as shown below and with this, we are ready to start our exercise of integrating Azure Synapse with Azure Purview.

Configuring Azure Purview for integrating with Azure Synapse Analytics

Open the Azure Synapse Studio by clicking on the Open Synapse Studio link from the dashboard page of Azure Synapse Workspace. Click on the Manage blade and you will see Azure Purview (Preview) under the External connections section as shown below. This feature is still in Preview as of the draft of this article. This feature allows us to integrate Synapse with Purview.

As shown above, we need to start by connecting our Azure Purview account here. Click on the button named Connect to a Purview account. It would pop-up a screen as shown below. If you have the Azure Purview account under the same Azure subscription in which the Azure Synapse Analytics account is created, when you select the “From Azure Subscription”, you will find the Purview account name as shown below.

Select the purview account and click on the Apply button. This will register the account with Azure Synapse as well as integrate it with Purview. Once done, you will receive a successful registration confirmation as shown below.

The benefit of connecting Azure Synapse with Azure Purview is that we can access the data assets from the catalog right in Azure Synapse Studio, and also use this information to initiate different actions supported by Synapse. To start accessing the Purview catalog from Synapse Studio, navigate to the Data tab and click on the search bar at the top of the screen as shown below. There would be a drop-down in the search bar which would have two options – Workspace and Purview. Ensure to select Purview as shown below. Now we are ready to start searching the catalog for data assets.

Type a full or partial name of the database object that we intend to search as shown below, and it would show a list of database objects that match the search criteria. These search results should not be confused with the database objects hosted in the Synapse pools which are part of the Synapse Workspace. As we are searching in the Purview catalog, the result would consist of data assets held in the specific purview account instance only. If we want to search for items within the workspace, we need to select the Workspace option in the drop-down which would list search results of objects in Azure Synapse.

The results are divided into two panes – the filters pane and the results pane. The filters pane shows the data asset type, classification and other such filters related to cataloged data assets. The results that meet the filter criteria as shown on the right pane. The results show the name of the data objects as well as the type of repository that holds the data object and address of the same.

Let’s say that we intend to explore the details of a particular data asset to understand whether it is suitable to be used as a source of data for data warehousing. We can click on the item in the results pane and it would show the results as shown below. In this case, it’s an Azure SQL Database table, so the details like Schema, Lineage, Data Classification, Related database objects, etc. are shown. On the right side of this screen, we can find the hierarchy under which this database object belongs.

Another interesting and useful feature of these results can be found in the related tab. At times, we may be searching for a database object but that may not be the exact match. Finding objects that are similar or related to the object being search can elevate the possibilities of finding the database object of interest. The related tab shows database objects like database, schemas, tables, or view depending on the hierarchy selected as shown below.

Once the data object of interest has been discovered, the next step is to take corresponding actions like creating a linked service, integration dataset, or a new data flow to source the data from the corresponding data repository. The Connect and Develop menu item provide links to initiate such actions as shown below. Clicking on these links would open a new pop-up window or wizard which would have the details of the data source and the data object already pre-populated. We can provide the credentials, build the corresponding artifact in Azure Synapse, and start sourcing the data from the targeted object.

The benefit of this integration is that we do not need to switch between two sets of services, gain access to the catalog which may be maintained by a data steward or data quality team, and port details back and forth from Azure Purview to Azure Synapse. The built-in integration eliminates all this overhead and provides the convenience of a catalog right within the operational console of a data warehousing environment.

#azure #sql azure #azure purview #azure synapse analytics

Azure Synapse Analytics Database CI/CD using Azure Function

In this article, I will discuss an Azure Database CI/CD approach using Azure Premium Function and Jenkins pipeline. I will only explain the architecture and the approach I took to implement the Database CI/CD pipeline.

Problem Statement and Challenges

I was working on a project where I had to build a Database deployment pipeline using enterprise GitHub which is only accessible through the company’s internal network. Also, port 1433 was blocked from the internal network to the Azure Synapse public endpoint for security reasons. Hence the only option I had was to run my pipeline in an internal network so that I could access GitHub which I was using for my Database Deployment Source Control and send the SQL code to Azure Synapse using Azure function HTTP post as port 1433 was blocked.

#azure #devops #azure-synapse-analytics #azure-devops #azure-functions