Evaluating the Potential and Promise of Machine Learning for UI Testing

Is it possible to automate UI testing using machine learning? This article takes a look at machine learning for automation testing and explores saving on manual labor of writing test cases and ideas for implementation.

Is it possible to automate UI testing using machine learning? This article takes a look at machine learning for automation testing and explores saving on manual labor of writing test cases and ideas for implementation.

Machine Learning Full Course - Learn Machine Learning

Machine Learning Full Course - Learn Machine Learning

This complete Machine Learning full course video covers all the topics that you need to know to become a master in the field of Machine Learning.

Machine Learning Full Course | Learn Machine Learning | Machine Learning Tutorial

It covers all the basics of Machine Learning (01:46), the different types of Machine Learning (18:32), and the various applications of Machine Learning used in different industries (04:54:48).This video will help you learn different Machine Learning algorithms in Python. Linear Regression, Logistic Regression (23:38), K Means Clustering (01:26:20), Decision Tree (02:15:15), and Support Vector Machines (03:48:31) are some of the important algorithms you will understand with a hands-on demo. Finally, you will see the essential skills required to become a Machine Learning Engineer (04:59:46) and come across a few important Machine Learning interview questions (05:09:03). Now, let's get started with Machine Learning.

Below topics are explained in this Machine Learning course for beginners:

  1. Basics of Machine Learning - 01:46

  2. Why Machine Learning - 09:18

  3. What is Machine Learning - 13:25

  4. Types of Machine Learning - 18:32

  5. Supervised Learning - 18:44

  6. Reinforcement Learning - 21:06

  7. Supervised VS Unsupervised - 22:26

  8. Linear Regression - 23:38

  9. Introduction to Machine Learning - 25:08

  10. Application of Linear Regression - 26:40

  11. Understanding Linear Regression - 27:19

  12. Regression Equation - 28:00

  13. Multiple Linear Regression - 35:57

  14. Logistic Regression - 55:45

  15. What is Logistic Regression - 56:04

  16. What is Linear Regression - 59:35

  17. Comparing Linear & Logistic Regression - 01:05:28

  18. What is K-Means Clustering - 01:26:20

  19. How does K-Means Clustering work - 01:38:00

  20. What is Decision Tree - 02:15:15

  21. How does Decision Tree work - 02:25:15 

  22. Random Forest Tutorial - 02:39:56

  23. Why Random Forest - 02:41:52

  24. What is Random Forest - 02:43:21

  25. How does Decision Tree work- 02:52:02

  26. K-Nearest Neighbors Algorithm Tutorial - 03:22:02

  27. Why KNN - 03:24:11

  28. What is KNN - 03:24:24

  29. How do we choose 'K' - 03:25:38

  30. When do we use KNN - 03:27:37

  31. Applications of Support Vector Machine - 03:48:31

  32. Why Support Vector Machine - 03:48:55

  33. What Support Vector Machine - 03:50:34

  34. Advantages of Support Vector Machine - 03:54:54

  35. What is Naive Bayes - 04:13:06

  36. Where is Naive Bayes used - 04:17:45

  37. Top 10 Application of Machine Learning - 04:54:48

  38. How to become a Machine Learning Engineer - 04:59:46

  39. Machine Learning Interview Questions - 05:09:03

Machine Learning for Automation Testing

Machine Learning for Automation Testing

The goals we are trying to achieve here by using Machine Learning for automation in testing are to dynamically write new test cases based on user interactions by data-mining their logs and their behavior on the application / service for which tests are to be written, live validation so that in case if an object is modified or removed or some other change like “modification in spelling” such as done by most of the IDE’s in the form of Intelli-sense like Visual Studio or Eclipse.

The goals we are trying to achieve here by using Machine Learning for automation in testing are to dynamically write new test cases based on user interactions by data-mining their logs and their behavior on the application / service for which tests are to be written, live validation so that in case if an object is modified or removed or some other change like “modification in spelling” such as done by most of the IDE’s in the form of Intelli-sense like Visual Studio or Eclipse.

Machine Learning in “Test Automation” can help prevent some of the following but not limited cases:

  1. Saving on Manual Labor of writing test cases
  2. Test cases are brittle so when something goes wrong a framework is most likely to either drop the testing at that point or to skip some steps which may result in wrong / failed result.
  3. Tests are not validated until and unless that test is run. So, if a script is written to check for an “OK” button then we wouldn’t know about its existence until we run the test.

The Machine Shall help recover from tests on the fly by applying fuzzy matching, that means if an object gets modified or removed then the program then the script must be able to find the closest object to the one it was looking for and then continue the test. For example, if a web services has options “small, medium, large” at first and the script was written according to that and if another choice i.e. “extra-large” is added then the script must be able to adapt to that and anticipate that change so that the test run can continue running without fail.

Know more about cross browser testing

HP Unified Function Testing is one of the well-known tools available in the market used for automated testing. It has a GUI interface for the same, other than that there are other tools like selenium libraries (implemented in several languages like Java, C++, C#, Python etc.) and Cucumber. Tools like these allow one to write their scripts and let the scripts take the job of testing from there by running through several cases.

Ideas for Implementation

Before starting the tests the system needs to learn the cases. We need to give it something to begin with. Before triggering the training part, we had to setup the system where some ads where shown to the normal users in a customized window of time, and during that time logs were collected and recorded which helped us in generating a gender ratio and age groups of people who looked at specific ads. The goal was to find out which age group and gender of users were stimulated to purchase something from the website after looking at certain ads presented to them. The results were saved as training data so that tests could be performed on them.

The machine was trained to write test cases based on the collected information. Training data was updated every once in a while so that tests could be run on the latest data based on different demographics and relevant ads could be delivered to potential customers. Had this been done manually one might have to make modifications in the script or add test cases manually each time trends changes or the website changes.

ML algorithms for automated testing

SVM: It belongs to the “linear classifier” family of ML algorithms that attempt to find a (linear) hyperplane that separates examples from different classes. In the learning phase, SVM treats the training data as a vector of k-1 dimensions. And the goal is to find maximum margin (distance). SVM technique is mostly used for the binary classification. Other than this we have MartiRank, a ranking algorithm, in the learning phase. It takes a number of rounds and during each round/iteration data is broken down into N sub-lists, each sub-list containing 1/n of the total number of device / app failures.

In case of Regression testing a suite of test cases needed to be developed, development of MartiRank is an ongoing process and has been used to detect new bugs. Ex: A dev might have refactored some code in the application and put into a new function. Regression testing showed us that the resulting models were different from the previous ones.

When we write test cases we test how the software is supposed to behave theoretically and there’s no real data with us, some of the test cases might never be used in real life and some that missed the test cases might be the most important ones and that is why data-mining the logs and letting the machine write test cases according to those logs automatically saves a lot of man-hours and helps in practical testing. Services like HockeyApp & TestFlight are providing automated mobile app testing as a service.

As for GUI tests there are some research papers out there that talk about DeepLearning and ReinforcementLearning for automation of the test. The systems that were being tested were first data-mined to get the meaningful clicks, texts and button pushes on the GUI interface which generated a good amount of training data. That data was then used to perform tests on the software for a few hours. Best part was that there were no need for models or test cases to be written and the bugs were being found as the time passed by, but some of the cases were not being tested which can be due to lack of training data. The reinforcement approach improved the testing as they were running through multiple iterations.

Intel and Nvidia have been investing heavily in hardware solutions that can aid Deep Learning and related algorithms to achieve results more quickly. Moving from a mobile first world to an AI first world. We know that for testing a certain product whether that be a small calculator there can never be enough number of right test cases and that is why developers and testers are encouraged to write more and more test cases in order to make their product more stable. Paul Graham once suggested the used of Bayesian Filter for filtering out spam emails, thousands of emails were fed to the system and it was made to learn then tests were performed on that training data to make sure that the filter was fool-proof. Web crawlers move through different websites looking for 404 or other errors all the time, updating their indexes and updating their test cases in real time.

Originally Published at LambdaTest

Machine Learning vs Front-End Developers: Future of UI design

Machine Learning vs Front-End Developers: Future of UI design

What will happen to front-end developers once machine learning becomes the norm for web programming?

A well-designed desktop or mobile website doesn’t just appear overnight. In fact, you need to go through a long process before you can publish your website. From concept sketches to a production-ready web app can, the entire process can take many months. But what if there were an easier, faster, and more efficient way to code and test a website layout? Thanks to machine learning, specifically deep learning, that dream is now a reality. Experts predict that machine learning and AI technology will change the process of front-end design for web designers in just a few short years. Of course, the question then arises: what will happen to front-end developers once machine learning becomes the norm for web programming?

Deep Learning and Website Design

Deep neural networks are a subclass of machine learning models that can be trained to make inference from large datasets. A neural network learns to do this by analyzing both the data input and the corresponding output. These models are widely used to solve AI challenges such as computer vision and natural language processing. Similar models can be used to design interface elements from relevant data.

There are at least two types of data to train a neural network to learn front-end development. The first is GUI screenshots (pix2code). Which is basically the screen you’re looking at right now. And the second is hand-drawn sketches (sketch2code). Each screenshot and sketch would include its respective code. Once the network has analyzed the initial image, it then moves on to language modeling or figuring out how to code the image in HTML or CSS. In the case of hand-drawn sketches, any written material also goes through a text recognition step.

Once these steps are completed, the neural network can apply its knowledge of objects, object positions, and layouts to code basic front-end elements. Ultimately, the neural network will become capable of “understanding” the visual design at a more abstract level (i.e. the intent behind the code). Deep learning eats up a lot of computing power, but the results are profound – breaking down the barriers to design and streamlining the entire process.

With the millions of websites on the internet, there’s no shortage of training data available for GUI screenshots. And each day thousands of sites go live, ensuring that any deep neural network using this method has access to the most up-to-date design information. This advantage also extends to iOS and Android mobile design capabilities.

Use Cases

AI-based UX design is more than just the latest futurist prediction. Startups and brand names alike are already using the technology in their day-to-day operations. Copenhagen-based startup Uizard has even built an entire business around the concept of transforming images into front-end code for mobile and web apps. The company is the first to build a machine learning algorithm that reads a single UI image and designs platform-specific code. Uizard can turn hand-drawn wireframes and digital designs into elegant, customizable and production-ready code. It reduces the amount of time that it takes for companies, both big and small, to go from concept to a finished product.

But they’re not the only ones looking ahead to the future of front-end design. Airbnb, one of the most recognizable brands in tech, is also getting in on the action. Starting from the premise that “the time required to test an idea should be zero,” the company has created its own prototype for transforming sketches into workable design code. Using the 150 components within the Airbnb UI system, the prototype’s machine learning algorithm was able to successfully create design code, with just a small amount of intermediary code to enable browser renderings.

The company has also live-coded prototypes from whiteboard drawings, turned production code into design files for front-end/UI designers, and enabled component specifications for engineers from mocks. Airbnb was also able to use their own coded images with a variety of platforms, including React Native, iOS Native, and Android Native. Perhaps most impressive is the fact that the UI stayed the same regardless of what device it was displayed on – a possible solution to the much-loathed browser/device incompatibility issue.

The Future of Front-End Design

A day in the life of a front-end developer largely consisted of turning mockups into usable code. This process can be both tedious and time-consuming. But for most developers, it’s an inescapable reality of the current design process. So, it must be completed before they can move on to more interesting steps. Like building out a site or creating the interesting features that make it unique and user-friendly.

Combining machine learning with front-end design gives developers more time for creative tasks. Instead of spending countless hours trying to decipher sketches or figuring out the graphic design, development and executive teams can simply get down to work. By doing what they do best – design. Best of all, a more streamlined design process will improve how companies in this space operate. Since everyone, right from CEOs to interns, will have a better understanding of the design process. And would be able to quickly try out design ideas with an understanding of protocols. What’s more, time to production would be relatively predictable. Whether you’re a bootstrapped startup or a big enterprise with a custom web portal, there’s one thing you need to consider. The optimized allocation of resources, time management, and design budgets can all add up to serious financial savings and increased ROI.

On a final note

Applying machine learning to front-end design may seem far-fetched. But it’s clear that it has the potential to both revolutionize how companies design. This new process may require front-end developers to become familiar with machine learning and programming languages. Or at least learn to work with different applications. But it’s unlikely to replace the need for them in the near future. This is especially true in industries where the ability to think outside the box and create a truly innovative and personalized user experience is essential to maintaining a competitive advantage. In short, don’t rush to put front-end developers on the endangered species list.

Thanks for reading

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Further reading

A Complete Machine Learning Project Walk-Through in Python

Machine Learning: how to go from Zero to Hero

Machine Learning for Front-End Developers

Top 18 Machine Learning Platforms For Developers

10 Amazing Articles On Python Programming And Machine Learning