1598768301
DevOps Tools Training | DevOps Tools | What is GIT | What is Jenkins | Intellipaat
In this DevOps Tools Training Video you will learn about the various DevOps Tools and You will understand What is GIT, What is Jenkins and What is Devops, How does DevOps Life cycle works. More over this DevOps Tools Training Helps you in clearing your DevOps Interviews
#devopstoolstraining #devopstools #whatisgit #devopstraining #devopscertification
What is GEEK
Buddha Community
1600401600
Jenkins Is Getting Old — It’s Time to Move On
By far, Jenkins is the most adopted tool for continuous integration, owning nearly 50% of the market share. As so many developers are using it, it has excellent community support, like no other Jenkins alternative. With that, it has more than 1,500 plugins available for continuous integration and delivery purposes.
We love and respect Jenkins. After all, it’s the first tool we encountered at the beginning of our automation careers. But as things are rapidly changing in the automation field, Jenkins is** left behind with his old approach**. Even though many developers and companies are using it, most of them aren’t happy with it. Having used it ourselves on previous projects, we quickly became frustrated by its lack of functionality, numerous maintenance issues, dependencies, and scaling problems.
We decided to investigate if other developers face the same problems and quickly saw the need to create a tool ourselves. We asked some developers at last year’s AWS Summit in Berlin about this. Most of them told us that they chose Jenkins because it’s free in the first place. However, many of them expressed interest in trying to use some other Jenkins alternative.
#devops #continuous integration #jenkins #devops adoption #jenkins ci #jenkins pipeline #devops continuous integration #jenkins automation #jenkins scripts #old technology
1593744480
DevOps Rewind - 2 | Jenkins Pipeline Tutorial | DevOps Tools | DevOps Training
This Edureka Video on “Jenkins pipeline Tutorial” will help you understand the basic concepts of a Jenkins pipeline along with a practical demo.
#devops #training #tools #tutorial #pipeline #jenkins
1602401329
How to Extend your DevOps Strategy For Success in the Cloud?
DevOps and Cloud computing are joined at the hip, now that fact is well appreciated by the organizations that engaged in SaaS cloud and developed applications in the Cloud. During the COVID crisis period, most of the organizations have started using cloud computing services and implementing a cloud-first strategy to establish their remote operations. Similarly, the extended DevOps strategy will make the development process more agile with automated test cases.
According to the survey in EMEA, IT decision-makers have observed a 129%* improvement in the overall software development process when performing DevOps on the Cloud. This success result was just 81% when practicing only DevOps and 67%* when leveraging Cloud without DevOps. Not only that, but the practice has also made the software predictability better, improve the customer experience as well as speed up software delivery 2.6* times faster.
3 Core Principle to fit DevOps Strategy
If you consider implementing DevOps in concert with the Cloud, then the
below core principle will guide you to utilize the strategy.
- It is indispensable to follow a continuous process, including all stages from Dev to deploy with the help of auto-provisioning resources of the target platform.
- The team always keeps an eye on major and minor application changes that can typically appear within a few hours of development to operation. However, the support of unlimited resource provisioning is needed at the stage of deployment.
- Cloud or hybrid configuration can associate this process, but you must confirm that configuration should support multiple cloud brands like Microsoft, AWS, Google, any public and private cloud models.
Guide to Remold Business with DevOps and Cloud
Companies are now re-inventing themselves to become better at sensing the next big thing their customers need and finding ways with the Cloud based DevOps to get ahead of the competition.
#devops #devops-principles #azure-devops #devops-transformation #good-company #devops-tools #devops-top-story #devops-infrastructure
1686166020
PyTorch Unsupervised Sentiment Discovery
** DEPRECATED **
This repo has been deprecated. Please visit Megatron-LM for our up to date Large-scale unsupervised pretraining and finetuning code.
If you would still like to use this codebase, see our tagged releases and install required software/dependencies that was available publicly at that date.
PyTorch Unsupervised Sentiment Discovery
This codebase contains pretrained binary sentiment and multimodel emotion classification models as well as code to reproduce results from our series of large scale pretraining + transfer NLP papers: Large Scale Language Modeling: Converging on 40GB of Text in Four Hours and Practical Text Classification With Large Pre-Trained Language Models. This effort was born out of a desire to reproduce, analyze, and scale the Generating Reviews and Discovering Sentiment paper from OpenAI.
The techniques used in this repository are general purpose and our easy to use command line interface can be used to train state of the art classification models on your own difficult classification datasets.
This codebase supports mixed precision training as well as distributed, multi-gpu, multi-node training for language models (support is provided based on the NVIDIA APEx project). In addition to training language models, this codebase can be used to easily transfer and finetune trained models on custom text classification datasets.
For example, a Transformer language model for unsupervised modeling of large text datasets, such as the amazon-review dataset, is implemented in PyTorch. We also support other tokenization methods, such as character or sentencepiece tokenization, and language models using various recurrent architectures.
The learned language model can be transferred to other natural language processing (NLP) tasks where it is used to featurize text samples. The featurizations provide a strong initialization point for discriminative language tasks, and allow for competitive task performance given only a few labeled samples. For example, we consider finetuning our models on the difficult task of multimodal emotion classification based on a subset of the plutchik wheel of emotions.
Created by Robert Plutchik, this wheel is used to illustrate different emotions in a compelling and nuanced way. He suggested that there are 8 primary bipolar emotions (joy versus sadness, anger versus fear, trust versus disgust, and surprise versus anticipation) with different levels of emotional intensity. For our classification task we utilize tweets from the SemEval2018 Task 1E-c emotion classification dataset to perform multilabel classification of anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. This is a difficult task that suffers from real world classification problems such as class imbalance and labeler disagreement.
On the full SemEval emotion classification dataset we find that finetuning our model on the data achieves competitive state of the art performance with no additional domain-specific feature engineering.
Setup
Install
Install the sentiment_discovery package with python3 setup.py install in order to run the modules/scripts within this repo.
Python Requirements
At this time we only support python3.
- numpy
- pytorch (>= 0.4.1)
- pandas
- scikit-learn
- matplotlib
- unidecode
- sentencepiece
- seaborn
- emoji
Pretrained models
We've included our sentencepiece tokenizer model and vocab as a zip file:
- sentencepiece tokenizer [1MB]
We've included a transformer language model base as well as a 4096-d mlstm language model base. For examples on how to use these models please see our finetuning and transfer sections. Even though these models were trained with FP16 they can be used in FP32 training/inference.
- FP16 Transformer LM [311MB]
- FP16 mLSTM LM [169MB]
We've also included classifiers trained on a subset of SemEval emotions corresponding to the 8 plutchik emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, and trust):
- Finetuned Plutchik Transformer [673MN]
- Finetuned Plutchik mLSTM [433MB]
Lastly, we've also included already trained classification models for SST and IMDB binary sentiment classification:
- Finetuned SST Transformer [621MB]
- Transferred SST mLSTM [325MB]
- Transferred IMDB mLSTM [325MB]
To use classification models that reproduce results from our original large batch language modeling paper please use the following commit hash and set of models.
We did not include pretrained models leveraging ELMo. To reproduce our papers' results with ELMo, please see our available resources.
Each file has a dictionary containing a PyTorch state_dict consisting of a language model (lm_encoder keys) trained on Amazon reviews and a classifier (classifier key) as well as accompanying args necessary to run a model with that state_dict.
Data Downloads
In the ./data folder we've provided processed copies of the Binary Stanford Sentiment Treebank (Binary SST), IMDB Movie Review, and the SemEval2018 Tweet Emotion datasets as part of this repository. In order to train on the amazon dataset please download the "aggressively deduplicated data" version from Julian McAuley's original site. Access requests to the dataset should be approved instantly. While using the dataset make sure to load it with the --loose-json flag.
Usage
In addition to providing easily reusable code of the core functionalities (models, distributed, fp16, etc.) of this work, we also provide scripts to perform the high-level functionalities of the original paper:
- sentiment classification of input text
- unsupervised reconstruction/language modeling of a corpus of text (+ script for launching distributed workers)
- transfer of learned language model to perform sentiment analysis on a specified corpus
- sampling from language model to generate text (possibly of fixed sentiment) + heatmap visualization of sentiment in text
Classifying text
Classify an input csv/json using one of our pretrained models or your own. Performs classification on Binary SST by default. Output classification probabilities are saved to a .npy file
python3 run_classifier.py --load_model ama_sst.pt # classify Binary SST
python3 run_classifier.py --load_model ama_sst_16.pt --fp16 # run classification in fp16
python3 run_classifier.py --load_model ama_sst.pt --text-key <text-column> --data <path.csv> # classify your own dataset
See here for more documentation.
Training Language Models (+ Distributed/FP16 Training)
Train a language model on a csv/json corpus. By default we train a weight-normalized, 4096-d mLSTM, with a 64-d character embedding. This is the first step of a 2-step process to training your own sentiment classifier. Saves model to lang_model.pt by default.
python3 pretrain.py #train a large model on imdb
python3 pretrain.py --model LSTM --nhid 512 #train a small LSTM instead
python3 pretrain.py --fp16 --dynamic-loss-scale #train a model with fp16
python3 -m multiproc pretrain.py #distributed model training
python3 pretrain.py --data ./data/amazon/reviews.json --lazy --loose-json \ #train a model on amazon data
--text-key reviewText --label-key overall --optim Adam --split 1000,1,1
python3 pretrain.py --tokenizer-type SentencePieceTokenizer --vocab-size 32000 \ #train a model with our sentencepiece tokenization
--tokenizer-type bpe --tokenizer-path ama_32k_tokenizer.model
python3 pretrain.py --tokenizer-type SentencePieceTokenizer --vocab-size 32000 \ #train a transformer model with our sentencepiece tokenization
--tokenizer-type bpe --tokenizer-path ama_32k_tokenizer.model --model transformer \
--decoder-layers 12 --decoder-embed-dim 768 --decoder-ffn-embed-dim 3072 \
--decoder-learned-pos --decoder-attention-heads 8
bash ./experiments/train_mlstm_singlenode.sh #run our mLSTM training script on 1 DGX-1V
bash ./experiments/train_transformer_singlenode.sh #run our transformer training script on 1 DGX-1V
For more documentation of our language modeling functionality look here
In order to learn about our language modeling experiments and reproduce results see the training reproduction section in analysis.
For information about how we achieve numerical stability with FP16 training see our fp16 training analysis.
Sentiment Transfer
Given a trained language model, this script will featurize text from train, val, and test csv/json's. It then uses sklearn logistic regression to fit a classifier to predict sentiment from these features. Lastly it performs feature selection to try and fit a regression model to the top n most relevant neurons (features). By default only one neuron is used for this second regression.
python3 transfer.py --load mlstm.pt #performs transfer to SST, saves results to `<model>_transfer/` directory
python3 transfer.py --load mlstm.pt --neurons 5 #use 5 neurons for the second regression
python3 transfer.py --load mlstm.pt --fp16 #run model in fp16 for featurization step
bash ./experiments/run_sk_sst.sh #run transfer learning with mlstm on imdb dataset
bash ./experiments/run_sk_imdb.sh #run transfer learning with mlstm on sst dataset
Additional documentation of the command line arguments available for transfer can be found here
Classifier Finetuning
Given a trained language model and classification dataset, this script will build a classifier that leverages the trained language model as a text feature encoder. The difference between this script and transfer.py is that the model training is performed end to end: the loss from the classifier is backpropagated into the language model encoder as well. This script allows one to build more complex classification models, metrics, and loss functions than transfer.py. This script supports building arbitrary multilable, multilayer, and multihead perceptron classifiers. Additionally it allows using language modeling as an auxiliary task loss during training and multihead variance as an auxiliary loss during training. Lastly this script supports automatically selecting classification thresholds from validation performance. To measure validation performance this script includes more complex metrics including: f1-score, mathew correlation coefficient, jaccard index, recall, precision, and accuracy.
python3 finetune_classifier.py --load mlstm.pt --lr 2e-5 --aux-lm-loss --aux-lm-loss-weight .02 #finetune mLSTM model on sst (default dataset) with auxiliary loss
python3 finetune_classifier.py --load mlstm.pt --automatic-thresholding --threshold-metric f1 #finetune mLSTM model on sst and automatically select classification thresholds based on the validation f1 score
python3 finetune_classifier.py --tokenizer-type SentencePieceTokenizer --vocab-size 32000 \ #finetune transformer with sentencepiece on SST
--tokenizer-type bpe tokenizer-path ama_32k_tokenizer.model --model transformer --lr 2e-5 \
--decoder-layers 12 --decoder-embed-dim 768 --decoder-ffn-embed-dim 3072 \
--decoder-learned-pos --decoder-attention-heads 8 --load transformer.pt --use-final-embed
python3 finetune_classifier.py --automatic-thresholding --non-binary-cols l1 l2 l3 --lr 2e-5\ #finetune multilayer classifier with 3 classes and 4 heads per class on some custom dataset and automatically select classfication thresholds
--classifier-hidden-layers 2048 1024 3 --heads-per-class 4 --aux-head-variance-loss-weight 1. #`aux-head-variance-loss-weight` is an auxiliary loss to increase the variance between each of the 4 head's weights
--data <custom_train>.csv --val <custom_val>.csv --test <custom_test>.csv --load mlstm.pt
bash ./experiments/se_transformer_multihead.sh #finetune a multihead transformer on 8 semeval categories
See how to reproduce our finetuning experiments in the finetuning reproduction section of analysis.
Additional documentation of the command line arguments available for finetune_classifier.py can be found here
Analysis
Acknowledgement
A special thanks to our amazing summer intern Neel Kant for all the work he did with transformers, tokenization, and pretraining+finetuning classification models.
A special thanks to @csarofeen and @Michael Carilli for their help developing and documenting our RNN interface, Distributed Data Parallel model, and fp16 optimizer. The latest versions of these utilities can be found at the APEx github page.
Thanks to @guillitte for providing a lightweight pytorch port of openai's sentiment-neuron repo.
This project uses the amazon review dataset collected by J. McAuley
Thanks
Want to help out? Open up an issue with questions/suggestions or pull requests ranging from minor fixes to new functionality.
May your learning be Deep and Unsupervised.
Download details:
Author: NVIDIA
Source: https://github.com/NVIDIA/sentiment-discovery
License: View license
1598001060
50+ Useful DevOps Tools
The article comprises both very well established tools for those who are new to the DevOps methodology.
What Is DevOps?
The DevOps methodology, a software and team management approach defined by the portmanteau of Development and Operations, was first coined in 2009 and has since become a buzzword concept in the IT field.
DevOps has come to mean many things to each individual who uses the term as DevOps is not a singularly defined standard, software, or process but more of a culture. Gartner defines DevOps as:
“DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.”
As you can see from the above definition, DevOps is a multi-faceted approach to the Software Development Life Cycle (SDLC), but its main underlying strength is how it leverages technology and software to streamline this process. So with the right approach to DevOps, notably adopting its philosophies of co-operation and implementing the right tools, your business can increase deployment frequency by a factor of 30 and lead times by a factor of 8000 over traditional methods, according to a CapGemini survey.
The Right Tools for the Job
This list is designed to be as comprehensive as possible. The article comprises both very well established tools for those who are new to the DevOps methodology and those tools that are more recent releases to the market — either way, there is bound to be a tool on here that can be an asset for you and your business. For those who already live and breathe DevOps, we hope you find something that will assist you in your growing enterprise.
With such a litany of tools to choose from, there is no “right” answer to what tools you should adopt. No single tool will cover all your needs and will be deployed across a variety of development and Operational teams, so let’s break down what you need to consider before choosing what tool might work for you.
- Plan and collaborate: Before you even begin the SDLC, your business needs to have a cohesive idea of what tools they’ll need to implement across your teams. There are even DevOps tools that can assist you with this first crucial step.
- Build: Here you want tools that create identically provisioned environments. The last you need is to hear “But it works for me on my computer”
- Automation: This has quickly become a given in DevOps, but automation will always drastically increase production over manual methods.
- Continuous Integration: Tools need to provide constant and immediate feedback, several times a day but not all integrations are implemented equally, will the tool you select be right for the job?
- Deployment: Deployments need to be kept predictable, smooth, and reliable with minimal risks, automation will also play a big part in this process.
With all that in mind, I hope this selection of tools will aid you as your business continues to expand into the DevOps lifestyle.
Tools Categories List:
Continuous Integration and Delivery
Infrastructure As Code
1. AWS CloudFormation
AWS CloudFormation is an absolute must if you are currently working, or planning to work, in the AWS Cloud. CloudFormation allows you to model your AWS infrastructure and provision all your AWS resources swiftly and easily. All of this is done within a JSON or YAML template file and the service comes with a variety of automation features ensuring your deployments will be predictable, reliable, and manageable.
Link: https://aws.amazon.com/cloudformation/
2. Azure Resource Manager
Azure Resource Manager (ARM) is Microsoft’s answer to an all-encompassing IAC tool. With its ARM templates, described within JSON files, Azure Resource Manager will provision your infrastructure, handle dependencies, and declare multiple resources via a single template.
Link: https://azure.microsoft.com/en-us/features/resource-manager/
3. Google Cloud Deployment Manager
Much like the tools mentioned above, Google Cloud Deployment Manager is Google’s IAC tool for the Google Cloud Platform. This tool utilizes YAML for its config files and JINJA2 or PYTHON for its templates. Some of its notable features are synchronistic deployment and ‘preview’, allowing you an overhead view of changes before they are committed.
Link: https://cloud.google.com/deployment-manager/
4. Terraform
Terraform is brought to you by HashiCorp, the makers of Vault and Nomad. Terraform is vastly different from the above-mentioned tools in that it is not restricted to a specific cloud environment, this comes with increased benefits for tackling complex distributed applications without being tied to a single platform. And much like Google Cloud Deployment Manager, Terraform also has a preview feature.
Link: https://www.terraform.io/
5. Chef
Chef is an ideal choice for those who favor CI/CD. At its heart, Chef utilizes self-described recipes, templates, and cookbooks; a collection of ready-made templates. Cookbooks allow for consistent configuration even as your infrastructure rapidly scales. All of this is wrapped up in a beautiful Ruby-based DSL pie.
Link: https://www.chef.io/products/chef-infra/
#tools #devops #devops 2020 #tech tools #tool selection #tool comparison