1679548821
Prompt engineering is a relatively new discipline for developing and optimizing prompts to efficiently use language models (LMs) for a wide variety of applications and research topics. Prompt engineering skills help to better understand the capabilities and limitations of large language models (LLMs). Researchers use prompt engineering to improve the capacity of LLMs on a wide range of common and complex tasks such as question answering and arithmetic reasoning. Developers use prompt engineering to design robust and effective prompting techniques that interface with LLMs and other tools.
Motivated by the high interest in developing with LLMs, we have created this new prompt engineering guide that contains all the latest papers, learning guides, lectures, references, and tools related to prompt engineering.
Happy Prompting!
The following are a set of guides on prompt engineering developed by us. Guides are work in progress.
If you are using the guide for your work, please cite us as follows:
@article{Saravia_Prompt_Engineering_Guide_2022,
author = {Saravia, Elvis},
journal = {https://github.com/dair-ai/Prompt-Engineering-Guide},
month = {12},
title = {{Prompt Engineering Guide}},
year = {2022}
}
Feel free to open a PR if you think something is missing here. Always welcome feedback and suggestions. Just open an issue!
We have published a 1 hour lecture that provides a comprehensive overview of prompting techniques, applications, and tools.
Author: Dair-ai
Source Code: https://github.com/dair-ai/Prompt-Engineering-Guide
License: MIT license
1673258580
a community-driven repository of examples using mlflow for tracking can be found at https://github.com/Azure/azureml-examples
Welcome to the Azure Machine Learning Python SDK notebooks repository!
These notebooks are recommended for use in an Azure Machine Learning Compute Instance, where you can run them without any additional set up.
However, the notebooks can be run in any development environment with the correct azureml
packages installed.
Install the azureml.core
Python package:
pip install azureml-core
Install additional packages as needed:
pip install azureml-mlflow
pip install azureml-dataset-runtime
pip install azureml-automl-runtime
pip install azureml-pipeline
pip install azureml-pipeline-steps
...
We recommend starting with one of the quickstarts.
This repository is a push-only mirror. Pull requests are ignored.
This project has adopted the Microsoft Open Source Code of Conduct. Please see the code of conduct for details.
Author: Azure
Source Code: https://github.com/Azure/MachineLearningNotebooks
License: MIT license
1672197626
With JupyterHub you can create a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server.
Project Jupyter created JupyterHub to support many users. The Hub can offer notebook servers to a class of students, a corporate data science workgroup, a scientific research project, or a high-performance computing group.
Three main actors make up JupyterHub:
Basic principles for operation are:
JupyterHub also provides a REST API for administration of the Hub and its users.
A Linux/Unix based system
Python 3.6 or greater
If you are using conda
, the nodejs and npm dependencies will be installed for you by conda.
If you are using pip
, install a recent version (at least 12.0) of nodejs/npm.
If using the default PAM Authenticator, a pluggable authentication module (PAM).
TLS certificate and key for HTTPS communication
Domain name
conda
To install JupyterHub along with its dependencies including nodejs/npm:
conda install -c conda-forge jupyterhub
If you plan to run notebook servers locally, install JupyterLab or Jupyter notebook:
conda install jupyterlab
conda install notebook
pip
JupyterHub can be installed with pip
, and the proxy with npm
:
npm install -g configurable-http-proxy
python3 -m pip install jupyterhub
If you plan to run notebook servers locally, you will need to install JupyterLab or Jupyter notebook:
python3 -m pip install --upgrade jupyterlab
python3 -m pip install --upgrade notebook
To start the Hub server, run the command:
jupyterhub
Visit http://localhost:8000
in your browser, and sign in with your system username and password.
Note: To allow multiple users to sign in to the server, you will need to run the jupyterhub
command as a privileged user, such as root. The wiki describes how to run the server as a less privileged user, which requires more configuration of the system.
The Getting Started section of the documentation explains the common steps in setting up JupyterHub.
The JupyterHub tutorial provides an in-depth video and sample configurations of JupyterHub.
To generate a default config file with settings and descriptions:
jupyterhub --generate-config
To start the Hub on a specific url and port 10.0.1.2:443
with https:
jupyterhub --ip 10.0.1.2 --port 443 --ssl-key my_ssl.key --ssl-cert my_ssl.cert
Authenticator | Description |
---|---|
PAMAuthenticator | Default, built-in authenticator |
OAuthenticator | OAuth + JupyterHub Authenticator = OAuthenticator |
ldapauthenticator | Simple LDAP Authenticator Plugin for JupyterHub |
kerberosauthenticator | Kerberos Authenticator Plugin for JupyterHub |
Spawner | Description |
---|---|
LocalProcessSpawner | Default, built-in spawner starts single-user servers as local processes |
dockerspawner | Spawn single-user servers in Docker containers |
kubespawner | Kubernetes spawner for JupyterHub |
sudospawner | Spawn single-user servers without being root |
systemdspawner | Spawn single-user notebook servers using systemd |
batchspawner | Designed for clusters using batch scheduling software |
yarnspawner | Spawn single-user notebook servers distributed on a Hadoop cluster |
wrapspawner | WrapSpawner and ProfilesSpawner enabling runtime configuration of spawners |
A starter docker image for JupyterHub gives a baseline deployment of JupyterHub using Docker.
Important: This jupyterhub/jupyterhub
image contains only the Hub itself, with no configuration. In general, one needs to make a derivative image, with at least a jupyterhub_config.py
setting up an Authenticator and/or a Spawner. To run the single-user servers, which may be on the same system as the Hub or not, Jupyter Notebook version 4 or greater must be installed.
The JupyterHub docker image can be started with the following command:
docker run -p 8000:8000 -d --name jupyterhub jupyterhub/jupyterhub jupyterhub
This command will create a container named jupyterhub
that you can stop and resume with docker stop/start
.
The Hub service will be listening on all interfaces at port 8000, which makes this a good choice for testing JupyterHub on your desktop or laptop.
If you want to run docker on a computer that has a public IP then you should (as in MUST) secure it with ssl by adding ssl options to your docker configuration or by using an ssl enabled proxy.
Mounting volumes will allow you to store data outside the docker image (host system) so it will be persistent, even when you start a new image.
The command docker exec -it jupyterhub bash
will spawn a root shell in your docker container. You can use the root shell to create system users in the container. These accounts will be used for authentication in JupyterHub's default configuration.
If you would like to contribute to the project, please read our contributor documentation and the CONTRIBUTING.md
. The CONTRIBUTING.md
file explains how to set up a development installation, how to run the test suite, and how to contribute to documentation.
For a high-level view of the vision and next directions of the project, see the JupyterHub community roadmap.
JupyterHub is supported on Linux/Unix based systems.
JupyterHub officially does not support Windows. You may be able to use JupyterHub on Windows if you use a Spawner and Authenticator that work on Windows, but the JupyterHub defaults will not. Bugs reported on Windows will not be accepted, and the test suite will not run on Windows. Small patches that fix minor Windows compatibility issues (such as basic installation) may be accepted, however. For Windows-based systems, we would recommend running JupyterHub in a docker container or Linux VM.
Additional Reference: Tornado's documentation on Windows platform support
We encourage you to ask questions and share ideas on the Jupyter community forum. You can also talk with us on our JupyterHub Gitter channel.
JupyterHub follows the Jupyter Community Guides.
Technical Overview | Installation | Configuration | Docker | Contributing | License | Help and Resources
Please note that this repository is participating in a study into the sustainability of open source projects. Data will be gathered about this repository for approximately the next 12 months, starting from 2021-06-11.
Data collected will include the number of contributors, number of PRs, time taken to close/merge these PRs, and issues closed.
For more information, please visit our informational page or download our participant information sheet.
Author: jupyterhub
Source Code: https://github.com/jupyterhub/jupyterhub
License: View license
1660288800
Pairwise Alignment and Dotplot ()
# # Pairwise Alignment and Dotplot
#
# **Requirements**:
#
# - Julia 1.6+
# - BioAlignments v2
# - BioSequences v2
#
# Pairwise sequence alignment algorithms are provided by the `BioAlignments.jl` package.
# The sequence alignment is a kind of optimization problem to find the best scoring alignment.
# For example, matching two similar symbols gets a high (often positive) score
# and matching dissimilar symbols does a low (often negative) score.
using BioSequences
using BioAlignments
# The following two sequences are homologus protein sequences from [titin](https://en.wikipedia.org/wiki/Titin)
# of human and mouse.
# Titin is a component of muscle fiber and extremely long (~30,000 amino acids)
# compared to other protein sequences.
# So, the sequences used here are just part of them.
# The complete sequences can be found here:
#
# - TITIN_HUMAN: <http://www.uniprot.org/uniprot/Q8WZ42>
# - TITIN_MOUSE: <http://www.uniprot.org/uniprot/A2ASS6>
titin_human = aa"""
EYTLLLIEAFPEDAAVYTCEAKNDYGVATTSASLSVEVPEVVSPDQEMPVYPPAIITPLQDTVTSEGQPA
RFQCRVSGTDLKVSWYSKDKKIKPSRFFRMTQFEDTYQLEIAEAYPEDEGTYTFVASNAVGQVSSTANLS
LEAPESILHERIEQEIEMEMKEFSSSFLSAEEEGLHSAELQLSKINETLELLSESPVYPTKFDSEKEGTG
PIFIKEVSNADISMGDVATLSVTVIGIPKPKIQWFFNGVLLTPSADYKFVFDGDDHSLIILFTKLEDEGE
YTCMASNDYGKTICSAYLKINSKGEGHKDTETESAVAKSLEKLGGPCPPHFLKELKPIRCAQGLPAIFEY
TVVGEPAPTVTWFKENKQLCTSVYYTIIHNPNGSGTFIVNDPQREDSGLYICKAENMLGESTCAAELLVL
LEDTDMTDTPCKAKSTPEAPEDFPQTPLKGPAVEALDSEQEIATFVKDTILKAALITEENQQLSYEHIAK
ANELSSQLPLGAQELQSILEQDKLTPESTREFLCINGSIHFQPLKEPSPNLQLQIVQSQKTFSKEGILMP
EEPETQAVLSDTEKIFPSAMSIEQINSLTVEPLKTLLAEPEGNYPQSSIEPPMHSYLTSVAEEVLSPKEK
TVSDTNREQRVTLQKQEAQSALILSQSLAEGHVESLQSPDVMISQVNYEPLVPSEHSCTEGGKILIESAN
PLENAGQDSAVRIEEGKSLRFPLALEEKQVLLKEEHSDNVVMPPDQIIESKREPVAIKKVQEVQGRDLLS
KESLLSGIPEEQRLNLKIQICRALQAAVA
"""
#-
titin_mouse = aa"""
EYTLLLIEAFPEDAAVYTCEAKNDYGVATTSASLSVEVPEVVSPDQEMPVYPPAIVTPLQDTVTSEGRPA
RFQCQVSGTDLKVSWYCKDKKIKPSRFFRMTQFEDTYQLEIAEAYPEDEGTYAFVANNAVGQVSSTATLR
LEAPESILHERIGQQIEMEMKEIASLLSAEEDFQTYSSDLRLPNANETLELLSEPPARSTQFDSRQEGAA
PVFIREISDVEISVEDVAKLSVTVTGCPKPKIQWFFNGMLLTPSADYKFVFDGDTHSLIILFTRFQDEGE
YTCLASNEYGKAVCSAHLRISPRGERSTEMESGEKKALEKPKGPCPPYFFKELKPVHCGPGIPAVFEYSV
HGEPAPTVLWFKEDMPLYTSVCYTIIHSPDGSGTFIVNDPQRGDSGLYLCKAQNLWGESTCAAELLVLPE
DTDVPDASCKEESTLGVPGDFLETSARGPLVQGVDSRQEITAFAEGTISKAALIAEETLQLSYERSVDDS
EVGTGVTIGAQKLPPVVLSTPQGTGELPSIDGAVHTQPGRGPPPTLNLQAVQAQTTLPKEATLQFEEPEG
VFPGASSAAQVSPVTIKPLITLTAEPKGNYPQSSTAAPDHALLSSVAAETLQLGEKKIPEVDKAQRALLL
SQSLAEGCVESLEVPDVAVSNMRSEPQVPFQHTCTEGKILMASADTLKSTGQDVALRTEEGKSLSFPLAL
EEKQVLLKEEQSEVVAVPTSQTSKSEKEPEAIKGVKEVREQELLSKETLFPSMPEEQRLHLKTQVRRALQ
AAVA
"""
# The similarity score between two symbols are described by a [substitution matrix](https://en.wikipedia.org/wiki/Substitution_matrix)
# and `BioAlignments.jl` provides some predefined matrices commonly seen in bioinformatics.
# `BLOSUM62` is a substitution matrix for amino acids
# and sometimes used as a default matrix in some programs.
# The derivation method and the biological rationale are outlined in this [wikipedia article](https://en.wikipedia.org/wiki/BLOSUM).
BLOSUM62
# The score in the matrix can be accessed using a pair of amino acids:
BLOSUM62[AA_R, AA_K]
# Two protein sequences are aligned to each other using the `pairalign` function,
# which takes an alignment optimization problem (`GlobalAlignment()`),
# two compared sequences (`titin_human`, `titin_mouse`) and scoring model (`model`).
# The algorithms implemented in this module can find the optimal sequence alignments
# between two sequences based on affine gap scoring model,
# in which inserting gaps of length `k` in a sequence alignment
# gets `gap_open + gap_extend * k` where `gap_open` and `gap_extend`
# are non-positive scores imposed on starting a new gap and extending a gap, respectively.
## Create a socring model from a substitution matrix and gap scores.
model = AffineGapScoreModel(BLOSUM62, gap_open=-11, gap_extend=-1)
#-
## Solve a pairwise-alignment optimization problem.
result = pairalign(GlobalAlignment(), titin_human, titin_mouse, model)
# The achieved score of the alignment is then:
score(result)
# The global alignment demonstrated above tries to align two sequences in an end-to-end manner.
# However, a researcher may be interested in local similarities between two sequences.
# In such a case, a [dot plot](https://en.wikipedia.org/wiki/Dot_plot_(bioinformatics))
# is a good visualization to detect local similarities.
using CairoMakie
"""
dotplot(seq1, seq2, submat; windowsize=20, threshold=20)
Create a dot plot from `seq1` and `seq2` based on `submat`;
`windowsize` specifies the size of a window to accumulate similarity scores along two sequences and
`threshold` specifies the score threshold to plot a white dot in a window (when `threshold` is NaN,
it plots raw similarity scores).
"""
function dotplot(seq1, seq2, submat; windowsize=20, threshold=20)
m = length(seq1)
n = length(seq2)
simscores = zeros(m - windowsize + 1, n - windowsize + 1)
for i in 1:m-windowsize+1, j in 1:n-windowsize+1
for k in 1:windowsize
simscores[i,j] += submat[seq1[i+k-1],seq2[j+k-1]]
end
end
fig = Figure(; resolution=(800,800))
if isnan(threshold)
colorscale = :viridis
ax = Axis(fig[1,1]; title="dotplot (windowsize=$windowsize, raw similarity score)")
else
colorscale = :greys
ax = Axis(fig[1,1]; title="dotplot (windowsize=$windowsize, threshold=$threshold)")
for i in eachindex(simscores)
simscores[i] = ifelse(simscores[i] > threshold, 1, 0)
end
end
heatmap!(simscores, colorscale=colorscale)
return fig
end
# We can see a raw similarity score plot by setting `threshold` to `NaN`.
dotplot(titin_human, titin_mouse, BLOSUM62, threshold=NaN)
# Filtering by the accumulated score highlights similar regions by white dots.
# The diagonal breaking lines (from the left-down to the right-up corner)
# correspond to the global similarity
# between the titin sequences of human and mouse as shown above,
# and other shorter lines around it
# corerspond to local similarities that are not captured in the global alignment.
dotplot(titin_human, titin_mouse, BLOSUM62)
RNA-Seq Coverage (outdated)
# RNA-Seq Coverage
#
# **Requirements**:
#
# - Julia 1.6+
# - BioSequences.jl v2
# - BioAlignments.jl v2
# - Xam.jl
# - ProgressMeters.jl
#
# In this tutorial we uses a public RNA-Seq data downloaded from the Sequence Read Archive (SRA):
# http://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR1238088.
# Paired-end FASTQ sequence files were extracted from the archive file
# and aligned to a reference genome using a couple of non-julia programs
#
# ... TODO
The programs under this repository are distributed in the public domain unless otherwise specified.
Author: BioJulia
Source Code: https://github.com/BioJulia/BioTutorials
1636132920
In this Coding Tutorial, We'll learn how to use Jupyter Notebook within our Terminal / Shell / Bash. Jupyter Notebook is usually available on the web browser but thanks to `nbterm` open sourced by David Brochart (sponsored by QuantStack) you can Use Jupyter Notebook from the Terminal / Shell /Bash with a similar rich experience like the web UI.
https://github.com/davidbrochart/nbterm
#python #data-science #workflow #terminal #jupyter #notebooks
1636106460
In this Python tutorial, We'll learn how to use imagededup, a Python library to find out duplicate images (using CNN - Convolutional Neural Networks). This tutorial doesn't require you to understand or know Neural Networks or Deep Learning. In just a few lines of Python Code, We'd have a Deep Learning based Image Deduping Script. For this demonstration, We'll try to identify Duplicate Pokemons.s
github - https://github.com/idealo/imagededup
1636087680
This Video Tutorial explains step-by-step guide of the Colab Notebook Hugging Face Notebook has put together to Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers. ASR - Automatic Speech Recognition. We demo with Hindi dataset from Mozilla Common Voice.
1636064760
In this Jupyter Notebook Tutorial, We'll learn about `jupyternotify` a Python package that lets us send Push Notifications from Jupyter Notebook after Code Execution. This is extremely helpful in Machine Learning or Deep Learning Model training to get to know about the completion of the training process or the progress of the Model training process.
1633804380
In this Programming tutorial, You'll learn how to get Native Jupyter Notebook experience on Visual Studio Code Insiders. Native Notebook on VSCode supports all the languages that your current Jupyter Notebook supports.
How to get Native Jupyter Notebook on VSCode
Step 1. Install VS Code Insiders
Step 2 If not working with Python, make sure to have a Jupyter kernelspec that corresponds to the language you would like to use installed on your machine.
Step 3. Install the Jupyter Extension
Step 4. Open or create a notebook file and start coding!
1633795380
Microsoft has refactored its VSCode Jupyter Extension to support all Jupyter Noteboook Kernels.That means, You can use languages other than Python on your VSCode Jupyter Notebook. Languages like R and Julia. In this Video you'll learn how to use R and Julia on VSCode Jupyter Notebook
1632567600
Use Bokeh to create interactive visualizations displayed on Jupyter Notebook
1627032960
A deep dive into what Notebooks are and how to use them in VS Code.
#notebooks #vs code
1622191396
A lightweight Python robot simulator for JupyterLab, Notebooks, and other Python environments.
For the core operations, you will need to install just aitk.robots:
pip install aitk.robots
To use the Jupyter enhancements, you’ll also need the browser-based extensions. You can install those with:
jupyter labextension install @jupyter-widgets/jupyterlab-manager
If not in a conda environment, then you will also need to:
jupyter nbextension enable --py widgetsnbextension
For additional information, please see:
Author: ArtificialIntelligenceToolkit
The Demo/Documentation: View The Demo/Documentation
Download Link: Download The Source Code
Official Website: https://github.com/ArtificialIntelligenceToolkit/aitk.robots
#python #jupyterlab #notebooks #ai #artificial-intelligence
1621295100
In this article, we will learn how to analyze data hosted in Azure Cosmos DB using Notebooks.
Azure Cosmos DB is a NoSQL data repository that supports a variety of API types to consume the data hosted in it. Jupyter Notebooks is one of the popular methods to access data for almost any type of data repositories. These notebooks support various types of SDKs, programming languages and frameworks. Python is one of the most popular programming languages and is used for a variety of purposes including data analysis with libraries like pandas and numpy. These libraries can be very effective in processing schemaless data structures that are typically stored in Azure Cosmos DB. Let’s learn how we can analyze data hosted in Cosmos DB using Notebooks in this article.
It is assumed that an Azure Cosmos DB account is already in place. In case not, then create a new account by opening the service and clicking on the New button. While creating the account, it provides an option to enable notebooks, which is Off by default. Ensure to switch “On” this option, as this will enable notebooks once the account is created. If you already have an account created, and this option was not turned on, on the console we would get an option to enable the notebooks. So, either way, ensure that the notebook feature is turned on. This feature is in preview as of the draft of this article, so it’s not yet recommended for use in a production environment till it goes generally available.
Once you have the account created with this feature turned on, when you open the account it would look as shown below. Navigate to the data explorer menu option, and under the Notebooks section, you would be able to see a gallery of notebooks as well as notebooks created by the end-user. In a new account, we would neither have any data nor any notebooks. We would create both in this exercise. Once we have an account that looks like this, our pre-requisite to start this exercise is met.
#azure #notebooks #azure cosmos db
1598068260
We are back with another episode of This Week on Channel 9! Join us as Christina, clad in a Ben Gibbord classic, catches you up on the latest developer news.
Please leave a comment or email us at twc9@microsoft.com.
#azure platform #azure services #notebooks #github