How To Migrate From Hadoop On-Premise To Microsoft Azure

In the past few months, cloud computing platforms have seen skyrocketing growth in the wake of the pandemic. The cloud market is estimated to grow from $371.4 billion from 2020 to $832.1 billion in 2025, at a CAGR of 17.5%. Meanwhile, tech giants have been working to make cloud systems more affordable and accessible to startups, application developers and enterprises.

The number of job openings in the cloud sector, ranging from cloud network engineer to cloud developer to cloud architect, has increased exponentially compared to the last two years. Previously, we discussed the steps to migrate from Hadoop on-premise to Google Cloud Platform (GCP).

Here, we will explain how to move the data from Hadoop on-premise to Microsoft Azure.

Benefits Of Migrating To Azure HDInsight

Before discussing the steps, let’s look at some of the benefits in migrating to Azure HDInsight.

• Easily Scalable: Azure HDInsight enables users to scale workloads up or down. The platform takes care of data redistribution as well as workload rebalancing without interrupting data processing jobs.
• Secure and Compliant: Azure HDInsight enables users to protect the enterprise data assets with encryption, and integration with Azure Active Directory.
• Simplified Version Management: Azure HDInsight manages the version of Hadoop ecosystem components and keeps them updated.
• **Extensibility With Custom Tools or Third-Party Applications: **The clusters of HDInsight can be extended with installed components. They can also be integrated with other popular big data solutions.
• Integration With Other Azure Services: HDInsight can easily be integrated with other popular Azure services, such as Azure Data Lake Storage Gen2, Azure SQL Database, Azure Data Factory (ADF), and more.
• Self-Healing Processes and Components: The Azure HDInsight monitors infrastructure and open-source components round the clock. It also recovers critical failures such as unavailability of open-source components and nodes automatically.

Steps To Migrate

Primarily, you have two key options to migrate data from Hadoop on-premises to Azure HDInsight:

1| Transfer data over network with TLS

• Over Internet- You can easily transfer large amounts of data to Azure cloud storage through the internet. This can be done by using several tools, including Azure Storage Explorer, AzCopy, Azure Powershell, etc.

#opinions #azure migration #hadoop to cloud #microsoft azure

What is GEEK

Buddha Community

How To Migrate From Hadoop On-Premise To Microsoft Azure

In the past few months, cloud computing platforms have seen skyrocketing growth in the wake of the pandemic. The cloud market is estimated to grow from $371.4 billion from 2020 to $832.1 billion in 2025, at a CAGR of 17.5%. Meanwhile, tech giants have been working to make cloud systems more affordable and accessible to startups, application developers and enterprises.

The number of job openings in the cloud sector, ranging from cloud network engineer to cloud developer to cloud architect, has increased exponentially compared to the last two years. Previously, we discussed the steps to migrate from Hadoop on-premise to Google Cloud Platform (GCP).

Here, we will explain how to move the data from Hadoop on-premise to Microsoft Azure.

Benefits Of Migrating To Azure HDInsight

Before discussing the steps, let’s look at some of the benefits in migrating to Azure HDInsight.

• Easily Scalable: Azure HDInsight enables users to scale workloads up or down. The platform takes care of data redistribution as well as workload rebalancing without interrupting data processing jobs.
• Secure and Compliant: Azure HDInsight enables users to protect the enterprise data assets with encryption, and integration with Azure Active Directory.
• Simplified Version Management: Azure HDInsight manages the version of Hadoop ecosystem components and keeps them updated.
• **Extensibility With Custom Tools or Third-Party Applications: **The clusters of HDInsight can be extended with installed components. They can also be integrated with other popular big data solutions.
• Integration With Other Azure Services: HDInsight can easily be integrated with other popular Azure services, such as Azure Data Lake Storage Gen2, Azure SQL Database, Azure Data Factory (ADF), and more.
• Self-Healing Processes and Components: The Azure HDInsight monitors infrastructure and open-source components round the clock. It also recovers critical failures such as unavailability of open-source components and nodes automatically.

Steps To Migrate

Primarily, you have two key options to migrate data from Hadoop on-premises to Azure HDInsight:

1| Transfer data over network with TLS

• Over Internet- You can easily transfer large amounts of data to Azure cloud storage through the internet. This can be done by using several tools, including Azure Storage Explorer, AzCopy, Azure Powershell, etc.

#opinions #azure migration #hadoop to cloud #microsoft azure

Migrating SQL workloads to Microsoft Azure: Planning the jump

In this article, we will discuss several points that should be considered when planning to migrate the on-premises SQL workload to Microsoft Azure cloud services. This article is the first step in a series of articles that discuss how to perform the SQL and No-SQL workload migration smoothly to the cloud.

Why migrate?

Data is one of the most precious “assets” in each company that drives business success. And as a proactive Data Engineer in an international company, you will always think how to secure your data at rest and in transit, and use the most optimal data platform technologies to serve the data to the application clients from any point in the earth as fast as possible with the minimum downtime or data loss possibilities.

With the growth of the company business, it is your responsibility to keep track of the data storage and retrieval speed in order not to lose your clients. Put yourself in the shoes of a client who is trying to submit an online order to buy from your online store, but the site is taking a long time to refresh the content and submit the order. For me, I will close the site and buy it from the nearest store!

If the Infrastructure administrator starts complaining about the limitation in the remaining resources in the current hosting machine, or the delay in receiving the new purchased resources, it is the suitable time to discuss with the management the choice to move your SQL workload to Microsoft Azure.

Initial Study

Before you think to send a meeting request to your management to discuss your idea about migrating the current SQL workload to Microsoft Azure, you need to take into consideration that it is not only one word that you need to mention to the management or a step by step tutorial that you can follow to perform the migration process. You should be prepared and ready for any question by preparing a comprehensive study that includes the current site problems and limitations, a plan for the design and implementation phases of the migration process, and the benefits that the company will gain from moving that workload to Azure from all performance, business growth handling and cost.

The initial study for the migration process should include, but may extend:

• The current SQL and No-SQL workload types in your company, such as OLTP and OLAP workloads
• The database administration and monitoring tools that you are using in the on-prems site
• The list of on-prems database engine types, versions, and locations
• The current database resources size and the expected resources growth ratio for each database
• The dependencies between the databases. This helps in selecting the list of candidate databases that will participate in the first migration wave and the database consolidation possibilities to reduce the cloud hosting cost
• The dependencies between the databases and the applications and how these applications interact with the databases. This will end up grouping the databases based on its dependencies. It is better to have a discussion with the development team to identify the criticality of each database for the business and see if it is possible to migrate the database applications to Microsoft Azure
• The different security and encryption requirements for each database
• The backup strategy and tools used for each database
• The list of all additional components that are involved in the data models, such as SSIS, SSAS, and SSRS that should be migrated
• The list of issues that you are facing in the current site, such as performance or availability
• The limitations in the current site, such as hardware upgrade limitation or unsupported features
• The Availability requirements for your databases
• The confirmed Restore Time Objective (RTO) and Restore Point Objective (RPO) for your databases and how they are met in the current site

Cloud Considerations

After checking the current situation, you should have a look at the cloud solutions that can be used to replace the current on-premises site. This includes learning the features available in each service and the pros and cons of each service in order to identify which service meets your requirements. With the different available cloud providers, I will concentrate on the Microsoft Azure database services in my articles, such as SQL on Azure VM, Azure SQL Database, Azure SQL Managed Instance, and Azure SQL Data Warehouse, and how to use it as a replacement for the on-premises ones.

Once you review the Microsoft Azure database-related services, your need to take into consideration the following points in your migration plan that may extend:

• Choose the suitable target data platform in the cloud, Infrastructure as a Services (IaaS) or Platform as a Service (PaaS), based on your technical requirements, and cost plan. This choice specifies what Azure services can be used and the control level on these services
• The Microsoft Azure services that can be used as a replacement for every single feature, component, or functionality in the on-prems site to serve your workloads
• The proper size for the Microsoft Azure services to fit the current site growth ratio with the minimal possible cost. Scaling up/down plans or scaling out/in plans at that stage will be a great step ahead
• The suitable region(s) to host each database that provides the lowest latency, based on the applications and client’s locations
• The Availability solution that should be used in Microsoft Azure, based on the selected Azure service, to meet your database availability requirements
• The cloud services configurations and features that can be used to meet the confirmed Restore Time Objective (RTO) and Restore Point Objective (RPO)
• The ability to achieve the security and privacy compliance and regulatory requirements of your organization using the cloud services
• The changes that should be performed on the current workload to be compatible with the target data platform technologies in Microsoft Azure and what transformation tools can be used to achieve that
• The new features in the cloud that can be used to optimize the current workload
• The administration and monitoring tools that can be used to replace the on-prems tools
• The ability to perform the security and encryption requirements for each database
• The backup and recovery cloud solutions can be used to design and automate the database backup strategy
• Any potential blockers for the migration process
• The tools that will be used to migrate the current workload to the cloud with minimal possible downtime and data loss
• Validation test strategy by migrating sample workload and how to measure the gains and compare it with the on-premises site
• The breaking fixes that should be performed after the migration process
• The rollback plan in case of any single possible failure

Now you can send the invitation to your management and discuss with them if migrating the current on-premises workload to the cloud is feasible, or you need to move with upgrading the current on-premises data center to handle the workload growth, serving the clients with highest possible availability and minimal data loss, without losing the company clients or having them frustrated from the company services.

#azure #migration #sql #sql-azure #sql-server #microsoft-azure

Migrating SQL workloads to Microsoft Azure: Services Selection

In the previous article, Migrating SQL workloads to Microsoft Azure: Planning the jump, we discussed the main points that should be checked and considered while drawing your plan to migrate the SQL workload from the on-premises datacenters to Microsoft Azure. In this article, we will go through the different database services that are provided by Microsoft Azure to help you in selecting the proper service that can serve your SQL workload when migrating it to Microsoft Azure.

IaaS or PaaS

Before choosing the suitable Microsoft Azure database service that meets your requirements, you need to specify the suitable Azure platform. Microsoft Azure provides two high-level platform options: Infrastructure as a Services, also known as IaaS and Platform as a Service, also known as PaaS. The platform choice specifies the Azure services that can be used and the control that you can have over the services under that platform.

Choosing the IaaS platform, you are renting the IT infrastructure servers and virtual machines from the cloud provider. This includes storage, networks, and operating systems. With this platform, you are still responsible for and have control over the Operating System layer and all layers over the OS, including the installation of the services, the operating system patching, and so on.

On the other hand, the PaaS platform provides you with the ability of building, testing, and deploying your applications without worrying about the underlying infrastructure management. In other words, you are not responsible for installing an operating system or patching the machine with the latest security and system updates.

The following image shows your responsibilities, in light blue, and the list of layers that you don’t need to worry about, in dark blue, where the cloud service provider, Microsoft for example, is responsible for managing the tasks fall under that layer. You can see that you are responsible for everything when hosting your databases in your datacenter, requiring multiple teams to handle these tasks, which is not possible for the start-up and small companies, as shown below:

Microsoft Azure Database Services

Now we are familiar with the difference between the platforms provided by Microsoft Azure. We need to identify the database services that are provided under each platform.

In the IaaS platform, you can rent a virtual machine and install your SQL Server instance in that machine, where you will be responsible for the Operating System and SQL Server installation and administration tasks under that service.

Moving to PaaS, you can see that Microsoft Azure provides you with different choices based on your workload type. For example, you can use Azure SQL Database or Azure SQL Managed Instance for your transactional SQL workload and use Azure Cosmos DB for your No-SQL transactional workload. For the analytical workload, you can use the Azure SQL Data Warehouse instance, under the Azure Synapse Analytics service.

Let us discuss each SQL database service provided by Microsoft Azure briefly.

SQL Server on Azure VM

The IaaS platform provides you with the ability to install and run your SQL Server instance in a fully managed Azure virtual machine. This option is the best choice when you plan to perform a lift-and-shift from your on-prem environment to Microsoft Azure with the minimal possible changes on your applications and databases schema, providing you with full control over the SQL Server instance and the Operating System management and security configurations, allowing you to host any number of user databases on that SQL Server VM, and provide you with the ability to configure customized high availability and disaster recovery solution.

SQL Server on Azure VM is suitable for you if your company already has IT teams to administrate that virtual machine from OS, networking, and security perspectives. And you will be billed for both the storage used to store your data and the compute operations consumed on that VM.

Rather than waiting for the purchase approval for the new hardware, you can easily, in a few minutes, deploy a new virtual machine in Azure, install a new SQL Server instance using your own license and connect to that SQL Server instance, with the ability to scale it up and down based on your requirements, and stop it during the idle time and resume it again when needed.

Azure SQL Database

Azure SQL Database, categorized under the PaaS platform, is a cloud-computing database service that provides you with the ability to host and use SQL databases in the cloud without worrying about the hardware and the software requirements. Although you are not responsible for the hardware security, the operating system patching and security, and the database files, that are encrypted at rest using the TDE feature, you are still responsible for preventing unauthorized access to the data by limiting the allowed IP addresses from the firewall side and the authorized users from the database access and permissions configuration.

Azure SQL Database provides us with many features, including the ability to automate the backup operation and keep your backup for up to 10 years, create a readable secondary replica to distribute the reporting workload to another datacenter, tune the performance automatically, Point-In-Time Restore, built-in high-availability, and the ability to scale the database resources on the fly up and down, by changing the Database Throughput Unit (DTU) value, and scale-out with no downtime, without the need to wait for any new hardware purchase order as in the on-prems scaling processes, as shown below:

By providing the name of the database and a few other options, your database will be up and running and ready to serve your transactional workload in a few minutes. With no hardware or operating system to buy or manage, you will pay only for what you use. Feel free to use the Azure Total Cost of Ownership Calculator to estimate the cost of your PaaS service usage.

Azure SQL Database can be deployed as a single database, purchased by DTU or vCore models, with its own set of resources managed by a logical SQL Server, that can be used when the database usage is stable. It can be also deployed using an elastic pool, purchased by eDTU or vCore models, that contains a group of databases that share the same set of resources and managed by a logical SQL Server, providing the best choice for the databases with frequently changing usage patterns. If your application surface area scoped at the database level, using the Azure SQL Database is the best choice.

#azure #migration #sql #sql-server #sql-azure #microsoft-azure

Securing Microsoft Active Directory

Clustering

K-means is one of the simplest unsupervised machine learning algorithms that solve the well-known data clustering problem. Clustering is one of the most common data analysis tasks used to get an intuition about data structure. It is defined as finding the subgroups in the data such that each data points in different clusters are very different. We are trying to find the homogeneous subgroups within the data. Each group’s data points are similarly based on similarity metrics like a Euclidean-based distance or correlation-based distance.

The algorithm can do clustering analysis based on features or samples. We try to find the subcategory of sampling based on attributes or try to find the subcategory of parts based on samples. The practical applications of such a procedure are many: the best use of clustering in amazon and Netflix recommended system, given a medical image of a group of cells, a clustering algorithm could aid in identifying the centers of the cells; looking at the GPS data of a user’s mobile device, their more frequently visited locations within a certain radius can be revealed; for any set of unlabeled observations, clustering helps establish the existence of some structure of data that might indicate that the data is separable.

What is K-Means Clustering?

K-means the clustering algorithm whose primary goal is to group similar elements or data points into a cluster.

K in k-means represents the number of clusters.

A cluster refers to a collection of data points aggregated together because of certain similarities.

K-means clustering is an iterative algorithm that starts with k random numbers used as mean values to define clusters. Data points belong to the group represented by the mean value to which they are closest. This mean value co-ordinates called the centroid.

Iteratively, the mean value of each cluster’s data points is computed, and the new mean values are used to restart the process till the mean stops changing. The disadvantage of k-means is that it a local search procedure and could miss global patterns.

The k initial centroids can be randomly selected. Another approach of determining k is to compute the entire dataset’s mean and add _k _random co-ordinates to it to make k initial points. Another method is to determine the principal component of the data and divide it into _k _equal partitions. The mean of each section can be used as initial centroids.

#ad #microsoft #microsoft-azure #azure #azure-functions #azure-security

What are the benefits of cloud migration? Reasons you should migrate

The moving of applications, databases and other business elements from the local server to the cloud server called cloud migration. This article will deal with migration techniques, requirement and the benefits of cloud migration.

In simple terms, moving from local to the public cloud server is called cloud migration. Gartner says 17.5% revenue growth as promised in cloud migration and also has a forecast for 2022 as shown in the following image.

#cloud computing services #cloud migration #all #cloud #cloud migration strategy #enterprise cloud migration strategy #business benefits of cloud migration #key benefits of cloud migration #benefits of cloud migration #types of cloud migration