Zara  Bryant

Zara Bryant

1617412761

AI Show Live-Episode 7-Run Azure Machine Learning anywhere

Join us this week to talk with Sauyra Das about this new Azure Machine Learning integration with Azure Arc – to enable Data Scientists to use existing kubernetes infrastructure on-premises or in multicloud to run machine learning. Stay tuned after as we get back to work on building our Roshambo game!

Join us on Learn TV for closed captioning https://aka.ms/LearnTV

Learn more:
Run Azure Machine Learning anywhere blog https://aka.ms/AIShow/AzureMLBlog
New Azure Arc Capabilities blog https://aka.ms/AIShow/AzureArcBlog

AI Show https://aka.ms/AIShow
Create a Free account (Azure) https://aka.ms/aishow-seth-azurefree
Get Started with Machine Learning https://aka.ms/AIShow/StartML
AI for Developers - https://aka.ms/AIShow/AIforDevelopers
Azure Machine Learning https://aka.ms/AIShow/AzureML

#azure #machine-learning #ai #artificial-intelligence

What is GEEK

Buddha Community

AI Show Live-Episode 7-Run Azure Machine Learning anywhere
Callum Slater

Callum Slater

1653465344

PySpark Cheat Sheet: Spark DataFrames in Python

This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples.

You'll probably already know about Apache Spark, the fast, general and open-source engine for big data processing; It has built-in modules for streaming, SQL, machine learning and graph processing. Spark allows you to speed analytic applications up to 100 times faster compared to other technologies on the market today. Interfacing Spark with Python is easy with PySpark: this Spark Python API exposes the Spark programming model to Python. 

Now, it's time to tackle the Spark SQL module, which is meant for structured data processing, and the DataFrame API, which is not only available in Python, but also in Scala, Java, and R.

Without further ado, here's the cheat sheet:

PySpark SQL cheat sheet

This PySpark SQL cheat sheet covers the basics of working with the Apache Spark DataFrames in Python: from initializing the SparkSession to creating DataFrames, inspecting the data, handling duplicate values, querying, adding, updating or removing columns, grouping, filtering or sorting data. You'll also see that this cheat sheet also on how to run SQL Queries programmatically, how to save your data to parquet and JSON files, and how to stop your SparkSession.

Spark SGlL is Apache Spark's module for working with structured data.

Initializing SparkSession 
 

A SparkSession can be used create DataFrame, register DataFrame as tables, execute SGL over tables, cache tables, and read parquet files.

>>> from pyspark.sql import SparkSession
>>> spark a SparkSession \
     .builder\
     .appName("Python Spark SQL basic example") \
     .config("spark.some.config.option", "some-value") \
     .getOrCreate()

Creating DataFrames
 

Fromm RDDs

>>> from pyspark.sql.types import*

Infer Schema

>>> sc = spark.sparkContext
>>> lines = sc.textFile(''people.txt'')
>>> parts = lines.map(lambda l: l.split(","))
>>> people = parts.map(lambda p: Row(nameap[0],ageaint(p[l])))
>>> peopledf = spark.createDataFrame(people)

Specify Schema

>>> people = parts.map(lambda p: Row(name=p[0],
               age=int(p[1].strip())))
>>>  schemaString = "name age"
>>> fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]
>>> schema = StructType(fields)
>>> spark.createDataFrame(people, schema).show()

 

From Spark Data Sources
JSON

>>>  df = spark.read.json("customer.json")
>>> df.show()

>>>  df2 = spark.read.load("people.json", format="json")

Parquet files

>>> df3 = spark.read.load("users.parquet")

TXT files

>>> df4 = spark.read.text("people.txt")

Filter 

#Filter entries of age, only keep those records of which the values are >24
>>> df.filter(df["age"]>24).show()

Duplicate Values 

>>> df = df.dropDuplicates()

Queries 
 

>>> from pyspark.sql import functions as F

Select

>>> df.select("firstName").show() #Show all entries in firstName column
>>> df.select("firstName","lastName") \
      .show()
>>> df.select("firstName", #Show all entries in firstName, age and type
              "age",
              explode("phoneNumber") \
              .alias("contactInfo")) \
      .select("contactInfo.type",
              "firstName",
              "age") \
      .show()
>>> df.select(df["firstName"],df["age"]+ 1) #Show all entries in firstName and age, .show() add 1 to the entries of age
>>> df.select(df['age'] > 24).show() #Show all entries where age >24

When

>>> df.select("firstName", #Show firstName and 0 or 1 depending on age >30
               F.when(df.age > 30, 1) \
              .otherwise(0)) \
      .show()
>>> df[df.firstName.isin("Jane","Boris")] #Show firstName if in the given options
.collect()

Like 

>>> df.select("firstName", #Show firstName, and lastName is TRUE if lastName is like Smith
              df.lastName.like("Smith")) \
     .show()

Startswith - Endswith 

>>> df.select("firstName", #Show firstName, and TRUE if lastName starts with Sm
              df.lastName \
                .startswith("Sm")) \
      .show()
>>> df.select(df.lastName.endswith("th"))\ #Show last names ending in th
      .show()

Substring 

>>> df.select(df.firstName.substr(1, 3) \ #Return substrings of firstName
                          .alias("name")) \
        .collect()

Between 

>>> df.select(df.age.between(22, 24)) \ #Show age: values are TRUE if between 22 and 24
          .show()

Add, Update & Remove Columns 

Adding Columns

 >>> df = df.withColumn('city',df.address.city) \
            .withColumn('postalCode',df.address.postalCode) \
            .withColumn('state',df.address.state) \
            .withColumn('streetAddress',df.address.streetAddress) \
            .withColumn('telePhoneNumber', explode(df.phoneNumber.number)) \
            .withColumn('telePhoneType', explode(df.phoneNumber.type)) 

Updating Columns

>>> df = df.withColumnRenamed('telePhoneNumber', 'phoneNumber')

Removing Columns

  >>> df = df.drop("address", "phoneNumber")
 >>> df = df.drop(df.address).drop(df.phoneNumber)
 

Missing & Replacing Values 
 

>>> df.na.fill(50).show() #Replace null values
 >>> df.na.drop().show() #Return new df omitting rows with null values
 >>> df.na \ #Return new df replacing one value with another
       .replace(10, 20) \
       .show()

GroupBy 

>>> df.groupBy("age")\ #Group by age, count the members in the groups
      .count() \
      .show()

Sort 
 

>>> peopledf.sort(peopledf.age.desc()).collect()
>>> df.sort("age", ascending=False).collect()
>>> df.orderBy(["age","city"],ascending=[0,1])\
     .collect()

Repartitioning 

>>> df.repartition(10)\ #df with 10 partitions
      .rdd \
      .getNumPartitions()
>>> df.coalesce(1).rdd.getNumPartitions() #df with 1 partition

Running Queries Programmatically 
 

Registering DataFrames as Views

>>> peopledf.createGlobalTempView("people")
>>> df.createTempView("customer")
>>> df.createOrReplaceTempView("customer")

Query Views

>>> df5 = spark.sql("SELECT * FROM customer").show()
>>> peopledf2 = spark.sql("SELECT * FROM global_temp.people")\
               .show()

Inspect Data 
 

>>> df.dtypes #Return df column names and data types
>>> df.show() #Display the content of df
>>> df.head() #Return first n rows
>>> df.first() #Return first row
>>> df.take(2) #Return the first n rows >>> df.schema Return the schema of df
>>> df.describe().show() #Compute summary statistics >>> df.columns Return the columns of df
>>> df.count() #Count the number of rows in df
>>> df.distinct().count() #Count the number of distinct rows in df
>>> df.printSchema() #Print the schema of df
>>> df.explain() #Print the (logical and physical) plans

Output

Data Structures 
 

 >>> rdd1 = df.rdd #Convert df into an RDD
 >>> df.toJSON().first() #Convert df into a RDD of string
 >>> df.toPandas() #Return the contents of df as Pandas DataFrame

Write & Save to Files 

>>> df.select("firstName", "city")\
       .write \
       .save("nameAndCity.parquet")
 >>> df.select("firstName", "age") \
       .write \
       .save("namesAndAges.json",format="json")

Stopping SparkSession 

>>> spark.stop()

Have this Cheat Sheet at your fingertips

Original article source at https://www.datacamp.com

#pyspark #cheatsheet #spark #dataframes #python #bigdata

Zara  Bryant

Zara Bryant

1617412761

AI Show Live-Episode 7-Run Azure Machine Learning anywhere

Join us this week to talk with Sauyra Das about this new Azure Machine Learning integration with Azure Arc – to enable Data Scientists to use existing kubernetes infrastructure on-premises or in multicloud to run machine learning. Stay tuned after as we get back to work on building our Roshambo game!

Join us on Learn TV for closed captioning https://aka.ms/LearnTV

Learn more:
Run Azure Machine Learning anywhere blog https://aka.ms/AIShow/AzureMLBlog
New Azure Arc Capabilities blog https://aka.ms/AIShow/AzureArcBlog

AI Show https://aka.ms/AIShow
Create a Free account (Azure) https://aka.ms/aishow-seth-azurefree
Get Started with Machine Learning https://aka.ms/AIShow/StartML
AI for Developers - https://aka.ms/AIShow/AIforDevelopers
Azure Machine Learning https://aka.ms/AIShow/AzureML

#azure #machine-learning #ai #artificial-intelligence

sophia tondon

sophia tondon

1620898103

5 Latest Technology Trends of Machine Learning for 2021

Check out the 5 latest technologies of machine learning trends to boost business growth in 2021 by considering the best version of digital development tools. It is the right time to accelerate user experience by bringing advancement in their lifestyle.

#machinelearningapps #machinelearningdevelopers #machinelearningexpert #machinelearningexperts #expertmachinelearningservices #topmachinelearningcompanies #machinelearningdevelopmentcompany

Visit Blog- https://www.xplace.com/article/8743

#machine learning companies #top machine learning companies #machine learning development company #expert machine learning services #machine learning experts #machine learning expert

Nora Joy

1604154094

Hire Machine Learning Developers in India

Hire machine learning developers in India ,DxMinds Technologies is the best product engineering company in India making innovative solutions using Machine learning and deep learning. We are among the best to hire machine learning experts in India work in different industry domains like Healthcare retail, banking and finance ,oil and gas, ecommerce, telecommunication ,FMCG, fashion etc.
**
Services**
Product Engineering & Development
Re-engineering
Maintenance / Support / Sustenance
Integration / Data Management
QA & Automation
Reach us 917483546629

Hire machine learning developers in India ,DxMinds Technologies is the best product engineering company in India making innovative solutions using Machine learning and deep learning. We are among the best to hire machine learning experts in India work in different industry domains like Healthcare retail, banking and finance ,oil and gas, ecommerce, telecommunication ,FMCG, fashion etc.

Services

Product Engineering & Development

Re-engineering

Maintenance / Support / Sustenance

Integration / Data Management

QA & Automation

Reach us 917483546629

#hire machine learning developers in india #hire dedicated machine learning developers in india #hire machine learning programmers in india #hire machine learning programmers #hire dedicated machine learning developers #hire machine learning developers

Nora Joy

1607006620

Applications of machine learning in different industry domains

Machine learning applications are a staple of modern business in this digital age as they allow them to perform tasks on a scale and scope previously impossible to accomplish.Businesses from different domains realize the importance of incorporating machine learning in business processes.Today this trending technology transforming almost every single industry ,business from different industry domains hire dedicated machine learning developers for skyrocket the business growth.Following are the applications of machine learning in different industry domains.

Transportation industry

Machine learning is one of the technologies that have already begun their promising marks in the transportation industry.Autonomous Vehicles,Smartphone Apps,Traffic Management Solutions,Law Enforcement,Passenger Transportation etc are the applications of AI and ML in the transportation industry.Following challenges in the transportation industry can be solved by machine learning and Artificial Intelligence.

  • ML and AI can offer high security in the transportation industry.
  • It offers high reliability of their services or vehicles.
  • The adoption of this technology in the transportation industry can increase the efficiency of the service.
  • In the transportation industry ML helps scientists and engineers come up with far more environmentally sustainable methods for powering and operating vehicles and machinery for travel and transport.

Healthcare industry

Technology-enabled smart healthcare is the latest trend in the healthcare industry. Different areas of healthcare, such as patient care, medical records, billing, alternative models of staffing, IP capitalization, smart healthcare, and administrative and supply cost reduction. Hire dedicated machine learning developers for any of the following applications.

  • Identifying Diseases and Diagnosis
  • Drug Discovery and Manufacturing
  • Medical Imaging Diagnosis
  • Personalized Medicine
  • Machine Learning-based Behavioral Modification
  • Smart Health Records
  • Clinical Trial and Research
  • Better Radiotherapy
  • Crowdsourced Data Collection
  • Outbreak Prediction

**
Finance industry**

In financial industries organizations like banks, fintech, regulators and insurance are Adopting machine learning to improve their facilities.Following are the use cases of machine learning in finance.

  • Fraud prevention
  • Risk management
  • Investment predictions
  • Customer service
  • Digital assistants
  • Marketing
  • Network security
  • Loan underwriting
  • Algorithmic trading
  • Process automation
  • Document interpretation
  • Content creation
  • Trade settlements
  • Money-laundering prevention
  • Custom machine learning solutions

Education industry

Education industry is one of the industries which is investing in machine learning as it offers more efficient and easierlearning.AdaptiveLearning,IncreasingEfficiency,Learning Analytics,Predictive Analytics,Personalized Learning,Evaluating Assessments etc are the applications of machine learning in the education industry.

Outsource your machine learning solution to India,India is the best outsourcing destination offering best in class high performing tasks at an affordable price.Business** hire dedicated machine learning developers in India for making your machine learning app idea into reality.
**
Future of machine learning

Continuous technological advances are bound to hit the field of machine learning, which will shape the future of machine learning as an intensively evolving language.

  • Improved Unsupervised Algorithms
  • Increased Adoption of Quantum Computing
  • Enhanced Personalization
  • Improved Cognitive Services
  • Rise of Robots

**Conclusion
**
Today most of the business from different industries are hire machine learning developers in India and achieve their business goals. This technology may have multiple applications, and, interestingly, it hasn’t even started yet but having taken such a massive leap, it also opens up so many possibilities in the existing business models in such a short period of time. There is no question that the increase of machine learning also brings the demand for mobile apps, so most companies and agencies employ Android developers and hire iOS developers to incorporate machine learning features into them.

#hire machine learning developers in india #hire dedicated machine learning developers in india #hire machine learning programmers in india #hire machine learning programmers #hire dedicated machine learning developers #hire machine learning developers