1640092029
Wafer Fault Detection with Python
Wafer Fault Detection
Problem Statement:
Wafer (In electronics), also called a slice or substrate, is a thin slice of semiconductor,
such as a crystalline silicon (c-Si), used for fabricationof integrated circuits and in photovoltaics,
to manufacture solar cells.
The inputs of various sensors for different wafers have been provided.
The goal is to build a machine learning model which predicts whether a wafer needs to be replaced or not
(i.e whether it is working or not) nased on the inputs from various sensors.
There are two classes: +1 and -1.
+1: Means that the wafer is in a working condition and it doesn't need to be replaced.
-1: Means that the wafer is faulty and it needa to be replaced.
Data Description
The client will send data in multiple sets of files in batches at a given location.
Data will contain Wafer names and 590 columns of different sensor values for each wafer.
The last column will have the "Good/Bad" value for each wafer.
Apart from training files, we laso require a "schema" file from the client, which contain all the
relevant information about the training files such as:
Name of the files, Length of Date value in FileName, Length of Time value in FileName, NUmber of Columnns,
Name of Columns, and their dataype.
Data Validation
In This step, we perform different sets of validation on the given set of training files.
Name Validation: We validate the name of the files based on the given name in the schema file. We have
created a regex patterg as per the name given in the schema fileto use for validation. After validating
the pattern in the name, we check for the length of the date in the file name as well as the length of time
in the file name. If all the values are as per requirements, we move such files to "Good_Data_Folder" else
we move such files to "Bad_Data_Folder."
Number of Columns: We validate the number of columns present in the files, and if it doesn't match with the
value given in the schema file, then the file id moves to "Bad_Data_Folder."
Name of Columns: The name of the columns is validated and should be the same as given in the schema file.
If not, then the file is moved to "Bad_Data_Folder".
The datatype of columns: The datatype of columns is given in the schema file. This is validated when we insert
the files into Database. If the datatype is wrong, then the file is moved to "Bad_Data_Folder."
Null values in columns: If any of the columns in a file have all the values as NULL or missing, we discard such
a file and move it to "Bad_Data_Folder".
Data Insertion in Database
Database Creation and Connection: Create a database with the given name passed. If the database is already created,
open the connection to the database.
Table creation in the database: Table with name - "Good_Data", is created in the database for inserting the files
in the "Good_Data_Folder" based on given column names and datatype in the schema file. If the table is already
present, then the new table is not created and new files are inserted in the already present table as we want
training to be done on new as well as old training files.
Insertion of file in the table: All the files in the "Good_Data_Folder" are inserted in the above-created table. If
any file has invalid data type in any of the columns, the file is not loaded in the table and is moved to
"Bad_Data_Folder".
Model Training
Data Export from Db: The data in a stored database is exported as a CSV file to be used for model training.
Data Preprocessing:
Check for null values in the columns. If present, impute the null values using the KNN imputer.
Check if any column has zero standard deviation, remove such columns as they don't give any information during
model training.
Clustering: KMeans algorithm is used to create clusters in the preprocessed data. The optimum number of clusters
is selected
Create a file "Dockerfile" with below content
FROM python:3.7
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
ENTRYPOINT [ "python" ]
CMD [ "main.py" ]
Create a "Procfile" with following content
web: gunicorn main:app
create a file ".circleci\config.yml" with following content
version: 2.1
orbs:
heroku: circleci/heroku@1.0.1
jobs:
build-and-test:
executor: heroku/default
docker:
- image: circleci/python:3.6.2-stretch-browsers
auth:
username: mydockerhub-user
password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
steps:
- checkout
- restore_cache:
key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
- run:
name: Install Python deps in a venv
command: |
echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
echo 'export IMAGE_NAME=python-circleci-docker' >> $BASH_ENV
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
- save_cache:
key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
paths:
- "venv"
- run:
command: |
. venv/bin/activate
python -m pytest -v tests/test_script.py
- store_artifacts:
path: test-reports/
destination: tr1
- store_test_results:
path: test-reports/
- setup_remote_docker:
version: 19.03.13
- run:
name: Build and push Docker image
command: |
docker build -t $DOCKERHUB_USER/$IMAGE_NAME:$TAG .
docker login -u $DOCKERHUB_USER -p $DOCKER_HUB_PASSWORD_USER docker.io
docker push $DOCKERHUB_USER/$IMAGE_NAME:$TAG
deploy:
executor: heroku/default
steps:
- checkout
- run:
name: Storing previous commit
command: |
git rev-parse HEAD > ./commit.txt
- heroku/install
- setup_remote_docker:
version: 18.06.0-ce
- run:
name: Pushing to heroku registry
command: |
heroku container:login
#heroku ps:scale web=1 -a $HEROKU_APP_NAME
heroku container:push web -a $HEROKU_APP_NAME
heroku container:release web -a $HEROKU_APP_NAME
workflows:
build-test-deploy:
jobs:
- build-and-test
- deploy:
requires:
- build-and-test
filters:
branches:
only:
- main
to create requirements.txt
pip freeze>requirements.txt
initialize git repo
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin <github_url>
git push -u origin main
create a account at circle ci
setup your project
Select project setting in CircleCI and below environment variable
DOCKERHUB_USER
DOCKER_HUB_PASSWORD_USER
HEROKU_API_KEY
HEROKU_APP_NAME
HEROKU_EMAIL_ADDRESS
DOCKER_IMAGE_NAME=wafercircle3270303
to update the modification
git add .
git commit -m "proper message"
git push
#docker login -u $DOCKERHUB_USER -p $DOCKER_HUB_PASSWORD_USER docker.io
Download Details:
Author: Avnish327030
Source Code: https://github.com/Avnish327030/wafer_circleci
License:
What is GEEK
Buddha Community
1619518440
top 30 Python Tips and Tricks for Beginners
Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.
1) swap two numbers.
2) Reversing a string in Python.
3) Create a single string from all the elements in list.
4) Chaining Of Comparison Operators.
5) Print The File Path Of Imported Modules.
6) Return Multiple Values From Functions.
7) Find The Most Frequent Value In A List.
8) Check The Memory Usage Of An Object.
…
#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners
1619510796
Lambda, Map, Filter functions in python
Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.
Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is
Syntax: x = lambda arguments : expression
Now i will show you some python lambda function examples:
#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map
1626775355
Why use Python for Software Development
No programming language is pretty much as diverse as Python. It enables building cutting edge applications effortlessly. Developers are as yet investigating the full capability of end-to-end Python development services in various areas.
By areas, we mean FinTech, HealthTech, InsureTech, Cybersecurity, and that's just the beginning. These are New Economy areas, and Python has the ability to serve every one of them. The vast majority of them require massive computational abilities. Python's code is dynamic and powerful - equipped for taking care of the heavy traffic and substantial algorithmic capacities.
Programming advancement is multidimensional today. Endeavor programming requires an intelligent application with AI and ML capacities. Shopper based applications require information examination to convey a superior client experience. Netflix, Trello, and Amazon are genuine instances of such applications. Python assists with building them effortlessly.
5 Reasons to Utilize Python for Programming Web Apps
Python can do such numerous things that developers can't discover enough reasons to admire it. Python application development isn't restricted to web and enterprise applications. It is exceptionally adaptable and superb for a wide range of uses.
Robust frameworks
Python is known for its tools and frameworks. There's a structure for everything. Django is helpful for building web applications, venture applications, logical applications, and mathematical processing. Flask is another web improvement framework with no conditions.
Web2Py, CherryPy, and Falcon offer incredible capabilities to customize Python development services. A large portion of them are open-source frameworks that allow quick turn of events.
Simple to read and compose
Python has an improved sentence structure - one that is like the English language. New engineers for Python can undoubtedly understand where they stand in the development process. The simplicity of composing allows quick application building.
The motivation behind building Python, as said by its maker Guido Van Rossum, was to empower even beginner engineers to comprehend the programming language. The simple coding likewise permits developers to roll out speedy improvements without getting confused by pointless subtleties.
Utilized by the best
Alright - Python isn't simply one more programming language. It should have something, which is the reason the business giants use it. Furthermore, that too for different purposes. Developers at Google use Python to assemble framework organization systems, parallel information pusher, code audit, testing and QA, and substantially more. Netflix utilizes Python web development services for its recommendation algorithm and media player.
Massive community support
Python has a steadily developing community that offers enormous help. From amateurs to specialists, there's everybody. There are a lot of instructional exercises, documentation, and guides accessible for Python web development solutions.
Today, numerous universities start with Python, adding to the quantity of individuals in the community. Frequently, Python designers team up on various tasks and help each other with algorithmic, utilitarian, and application critical thinking.
Progressive applications
Python is the greatest supporter of data science, Machine Learning, and Artificial Intelligence at any enterprise software development company. Its utilization cases in cutting edge applications are the most compelling motivation for its prosperity. Python is the second most well known tool after R for data analytics.
The simplicity of getting sorted out, overseeing, and visualizing information through unique libraries makes it ideal for data based applications. TensorFlow for neural networks and OpenCV for computer vision are two of Python's most well known use cases for Machine learning applications.
Summary
Thinking about the advances in programming and innovation, Python is a YES for an assorted scope of utilizations. Game development, web application development services, GUI advancement, ML and AI improvement, Enterprise and customer applications - every one of them uses Python to its full potential.
The disadvantages of Python web improvement arrangements are regularly disregarded by developers and organizations because of the advantages it gives. They focus on quality over speed and performance over blunders. That is the reason it's a good idea to utilize Python for building the applications of the future.
#python development services #python development company #python app development #python development #python in web development #python software development
1619799996
Deep Learning Project : Real Time Object Detection in Python & Opencv
Real Time Object Detection in Python And OpenCV
Github Link: https://github.com/Chando0185/Object_Detection
Blog Link: https://knowledgedoctor37.blogspot.com/#
I’m on Instagram as @knowledge_doctor.
Follow Me On Instagram :
https://www.instagram.com/invites/contact/?i=f9n3ongbu8ma&utm_content=jresydt
Like My Facebook Page:
https://www.facebook.com/Knowledge-Doctor-Programming-114082097010409/
#python project #object detection #python opencv #opencv object detection #object detection in python #python opencv for object detection
1602968400
Python Tricks Every Developer Should Know
Python is awesome, it’s one of the easiest languages with simple and intuitive syntax but wait, have you ever thought that there might ways to write your python code simpler?
In this tutorial, you’re going to learn a variety of Python tricks that you can use to write your Python code in a more readable and efficient way like a pro.
Let’s get started
Swapping value in Python
Instead of creating a temporary variable to hold the value of the one while swapping, you can do this instead
>>> FirstName = "kalebu"
>>> LastName = "Jordan"
>>> FirstName, LastName = LastName, FirstName
>>> print(FirstName, LastName)
('Jordan', 'kalebu')
#python #python-programming #python3 #python-tutorials #learn-python #python-tips #python-skills #python-development