1653278640
WSGI Types for Python
This is an attempt to bring some type safety to WSGI applications using Python's new typing features (TypedDicts, Protocols). It seems to work OK but may still be full of gaps, holes, bugs, missteps, etc. It helped bring a lot of extra safety to a couple of places that really needed it though, and seemed to remain quite stable for a couple of months.
This is implemented as a Python module, rather than MyPy stubs, as it represents a protocol things can satisfy rather than a set of types for something concrete.
This package came together during an exploration documented here: python/mypy#7654
Define a callable application as a class:
import wsgitypes
class MyApplication(wsgitypes.Application):
def __call__(
self,
environ: wsgitypes.Environ,
start_response: wsgitypes.StartResponse,
) -> wsgitypes.Response:
my_header = environ.get("REQUEST_METHOD", "")
return []
Environ should be type-safe:
class MyApplication(wsgitypes.Application):
def __call__(
self,
environ: wsgitypes.Environ,
start_response: wsgitypes.StartResponse,
) -> wsgitypes.Response:
environ["wsgi.input"] # Good
environ["wsgi.unpot"] # BORK! MyPy will catch this.
return []
You can define your own extensions to Environ
using TypedDict
inheritance, like so:
class MyEnviron(wsgitypes.Environ):
HTTP_X_MY_HEADER: t.Optional[str]
class MyApplication(wsgitypes.Application):
def __call__(
self,
environ: MyEnviron,
start_response: wsgitypes.StartResponse,
) -> wsgitypes.Response:
environ = typing.cast(MyEnviron, environ)
environ.get("HTTP_X_MY_HEADER") # Good
return []
Note that you need to use typing.cast
to convert the incoming Environ to your derived version. An attempt was made to use a type param for Environ, but it wasn't viable (even with GVR helping!): python/mypy#7654
Author: shabbyrobe
Source Code: https://github.com/shabbyrobe/wsgitypes
License: MIT license
1653278400
Python pandas tutorial on how to loop over columns or iterate over the columns using itermitems to get a value of single column or a specific row and apply any mathematical operation on that.
1653278193
In todays tutorial we will learn how to automate Twitter processes, or in other words - we will create an extra friendly liking bot! ๐ค๐ค๐ค
This live streaming tutorial is perfect for beginners! all you need is to have Python installed on your system (I'm using Python 3.9 in case you'd like to match versions ๐คช) and we will take care of everything else together!
Complete tutorial code can be found on my Github:
https://github.com/MariyaSha/TwitterBot
The main purpose of this stream is to reflect on how easy it is to set up bots on social media platforms. Additionally, we will discuss whether you think it's good or bad - are extra layers of protection required? or do you like the current "wild wild west" situation? ๐
And even though our bot is going to be fluffy and harmless - can you imagine what experienced black hats with malicious intents have been doing with this technology throughout the years? (hint: phishing, impersonating and other fraud-related issues)
So I propose this experiment - will Twitter ever detect our bots? and how soon will this happen? (if at all)
#python #twitter #chatbot
1653277912
This video included everything you need to get started with ArcGIS. We also added one real world GIS project like risk analysis in ArcGIS.
Table of Content
1.00 - where to download the software
1.30 - ArcGIS interface introduction
5.30 - Vector data vs raster data
8.00 - map template
10.00 - shape file creation
12.00 - map digitizing
23.00 - different ways of shape file editing
29.00 - label designing
36.00 - projection system change
37.00 - Area Calculation
40.00 - Saving as .MXD file ****************** Very important
43.00 - Geodatabase Design***************
01:2 - Topological Rule**********
1:11 - Study area mapping
1:34 - Different joins (merge, union, table join, excel sheet join)
1:56 - Different types of buffering
2:23 - Data processing (with little bit of SQL & Python commands)*********
2:37 - Raster data analysis
- Entire data frame clipping
- Clipping
- Extract by Mask
- NDVI
- IDW
- zonal stat
2:56 - Final Project
- IDW
- SRTM satellite data download (DEM)
- slope derivation
- Reclassification
- Rating
- Feature to raster
-weighted sum
- Final index
Dataset to Practice:
https://drive.google.com/drive/folders/1EwTVMmsrpYBK5ujmfeFAsmM4Wz2Who4s?usp=sharing
SUBSCRIBE: https://www.youtube.com/c/DataAnalyticsm/featured
1653275929
์์ฆ ์์ ผ ๋๋ฆฌ๋ 'ํ์ด์คํฌ๋ฆฝํธ' PyScript 10๋ถ ์ค๋ช
!
์๊ฐ~ ์๋ฐ์คํฌ๋ฆฝํธ~ ์ด๋๊น์ง ๊ณ ๋ง์ ๊ณ . ๋ค์ ๋ณด์ง๋ง์ (!?!?)
#pyscript #anaconda #python #javascript #html
1653273840
PyAutoGUI
PyAutoGUI is a cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.
pip install pyautogui
Full documentation available at https://pyautogui.readthedocs.org
Simplified Chinese documentation available at https://github.com/asweigart/pyautogui/blob/master/docs/simplified-chinese.ipynb
Source code available at https://github.com/asweigart/pyautogui
If you need help installing Python, visit https://installpython3.com/
Dependencies
PyAutoGUI supports Python 2 and 3. If you are installing PyAutoGUI from PyPI using pip:
Windows has no dependencies. The Win32 extensions do not need to be installed.
macOS needs the pyobjc-core and pyobjc module installed (in that order).
Linux needs the python3-xlib (or python-xlib for Python 2) module installed.
Pillow needs to be installed, and on Linux you may need to install additional libraries to make sure Pillow's PNG/JPEG works correctly. See:
https://stackoverflow.com/questions/7648200/pip-install-pil-e-tickets-1-no-jpeg-png-support
http://ubuntuforums.org/showthread.php?t=1751455
If you want to do development and contribute to PyAutoGUI, you will need to install these modules from PyPI:
Example Usage
The x, y coordinates used by PyAutoGUI has the 0, 0 origin coordinates in the top left corner of the screen. The x coordinates increase going to the right (just as in mathematics) but the y coordinates increase going down (the opposite of mathematics). On a screen that is 1920 x 1080 pixels in size, coordinates 0, 0 are for the top left while 1919, 1079 is for the bottom right.
Currently, PyAutoGUI only works on the primary monitor. PyAutoGUI isn't reliable for the screen of a second monitor (the mouse functions may or may not work on multi-monitor setups depending on your operating system and version).
All keyboard presses done by PyAutoGUI are sent to the window that currently has focus, as if you had pressed the physical keyboard key.
>>> import pyautogui
>>> screenWidth, screenHeight = pyautogui.size() # Returns two integers, the width and height of the screen. (The primary monitor, in multi-monitor setups.)
>>> currentMouseX, currentMouseY = pyautogui.position() # Returns two integers, the x and y of the mouse cursor's current position.
>>> pyautogui.moveTo(100, 150) # Move the mouse to the x, y coordinates 100, 150.
>>> pyautogui.click() # Click the mouse at its current location.
>>> pyautogui.click(200, 220) # Click the mouse at the x, y coordinates 200, 220.
>>> pyautogui.move(None, 10) # Move mouse 10 pixels down, that is, move the mouse relative to its current position.
>>> pyautogui.doubleClick() # Double click the mouse at the
>>> pyautogui.moveTo(500, 500, duration=2, tween=pyautogui.easeInOutQuad) # Use tweening/easing function to move mouse over 2 seconds.
>>> pyautogui.write('Hello world!', interval=0.25) # Type with quarter-second pause in between each key.
>>> pyautogui.press('esc') # Simulate pressing the Escape key.
>>> pyautogui.keyDown('shift')
>>> pyautogui.write(['left', 'left', 'left', 'left', 'left', 'left'])
>>> pyautogui.keyUp('shift')
>>> pyautogui.hotkey('ctrl', 'c')
>>> import pyautogui
>>> pyautogui.alert('This is an alert box.')
'OK'
>>> pyautogui.confirm('Shall I proceed?')
'Cancel'
>>> pyautogui.confirm('Enter option.', buttons=['A', 'B', 'C'])
'B'
>>> pyautogui.prompt('What is your name?')
'Al'
>>> pyautogui.password('Enter password (text will be hidden)')
'swordfish'
(PyAutoGUI uses Pillow for image-related features.)
>>> import pyautogui
>>> im1 = pyautogui.screenshot()
>>> im1.save('my_screenshot.png')
>>> im2 = pyautogui.screenshot('my_screenshot2.png')
You can also locate where an image is on the screen:
>>> import pyautogui
>>> button7location = pyautogui.locateOnScreen('button.png') # returns (left, top, width, height) of matching region
>>> button7location
(1416, 562, 50, 41)
>>> buttonx, buttony = pyautogui.center(button7location)
>>> buttonx, buttony
(1441, 582)
>>> pyautogui.click(buttonx, buttony) # clicks the center of where the button was found
The locateCenterOnScreen() function returns the center of this match region:
>>> import pyautogui
>>> buttonx, buttony = pyautogui.locateCenterOnScreen('button.png') # returns (x, y) of matching region
>>> buttonx, buttony
(1441, 582)
>>> pyautogui.click(buttonx, buttony) # clicks the center of where the button was found
How Does PyAutoGUI Work?
The three major operating systems (Windows, macOS, and Linux) each have different ways to programmatically control the mouse and keyboard. This can often involve confusing, obscure, and deeply technical details. The job of PyAutoGUI is to hide all of this complexity behind a simple API.
On Windows, PyAutoGUI accesses the Windows API (also called the WinAPI or win32 API) through the built-in ctypes
module. The nicewin
module at https://github.com/asweigart/nicewin provides a demonstration for how Windows API calls can be made through Python.
On macOS, PyAutoGUI uses the rubicon-objc
module to access the Cocoa API.
On Linux, PyAutoGUI uses the Xlib
module to access the X11 or X Window System.
If you find this project helpful and would like to support its development, consider donating to its creator on Patreon.
Author: asweigart
Source Code: https://github.com/asweigart/pyautogui
License: BSD-3-Clause License
1653271140
Python pandas series tutorial on creating multiple index or hierarchical index and processing it by applying series functions.
1653268320
Python pandas tutorial for beginners on how to rank dataframe values in python using the rank function and re order the dataframe.
1653265860
torchtyping
Turn this:
def batch_outer_product(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
# x has shape (batch, x_channels)
# y has shape (batch, y_channels)
# return has shape (batch, x_channels, y_channels)
return x.unsqueeze(-1) * y.unsqueeze(-2)
into this:
def batch_outer_product(x: TensorType["batch", "x_channels"],
y: TensorType["batch", "y_channels"]
) -> TensorType["batch", "x_channels", "y_channels"]:
return x.unsqueeze(-1) * y.unsqueeze(-2)
with programmatic checking that the shape (dtype, ...) specification is met.
Bye-bye bugs! Say hello to enforced, clear documentation of your code.
If (like me) you find yourself littering your code with comments like # x has shape (batch, hidden_state)
or statements like assert x.shape == y.shape
, just to keep track of what shape everything is, then this is for you.
pip install torchtyping
Requires Python 3.7+ and PyTorch 1.7.0+.
torchtyping
allows for type annotating:
...
;torchtyping
is highly extensible.If typeguard
is (optionally) installed then at runtime the types can be checked to ensure that the tensors really are of the advertised shape, dtype, etc.
# EXAMPLE
from torch import rand
from torchtyping import TensorType, patch_typeguard
from typeguard import typechecked
patch_typeguard() # use before @typechecked
@typechecked
def func(x: TensorType["batch"],
y: TensorType["batch"]) -> TensorType["batch"]:
return x + y
func(rand(3), rand(3)) # works
func(rand(3), rand(1))
# TypeError: Dimension 'batch' of inconsistent size. Got both 1 and 3.
typeguard
also has an import hook that can be used to automatically test an entire module, without needing to manually add @typeguard.typechecked
decorators.
If you're not using typeguard
then torchtyping.patch_typeguard()
can be omitted altogether, and torchtyping
just used for documentation purposes. If you're not already using typeguard
for your regular Python programming, then strongly consider using it. It's a great way to squash bugs. Both typeguard
and torchtyping
also integrate with pytest
, so if you're concerned about any performance penalty then they can be enabled during tests only.
torchtyping.TensorType[shape, dtype, layout, details]
The core of the library.
Each of shape
, dtype
, layout
, details
are optional.
shape
argument can be any of:int
: the dimension must be of exactly this size. If it is -1
then any size is allowed.str
: the size of the dimension passed at runtime will be bound to this name, and all tensors checked that the sizes are consistent....
: An arbitrary number of dimensions of any sizes.str: int
pair (technically it's a slice), combining both str
and int
behaviour. (Just a str
on its own is equivalent to str: -1
.)str: str
pair, in which case the size of the dimension passed at runtime will be bound to both names, and all dimensions with either name must have the same size. (Some people like to use this as a way to associate multiple names with a dimension, for extra documentation purposes.)str: ...
pair, in which case the multiple dimensions corresponding to ...
will be bound to the name specified by str
, and again checked for consistency between arguments.None
, which when used in conjunction with is_named
below, indicates a dimension that must not have a name in the sense of named tensors.None: int
pair, combining both None
and int
behaviour. (Just a None
on its own is equivalent to None: -1
.)None: str
pair, combining both None
and str
behaviour. (That is, it must not have a named dimension, but must be of a size consistent with other uses of the string.)typing.Any
: Any size is allowed for this dimension (equivalent to -1
).TensorType["batch": ..., "length": 10, "channels", -1]
. If you just want to specify the number of dimensions then use for example TensorType[-1, -1, -1]
for a three-dimensional tensor.dtype
argument can be any of:torch.float32
, torch.float64
etc.int
, bool
, float
, which are converted to their corresponding PyTorch types. float
is specifically interpreted as torch.get_default_dtype()
, which is usually float32
.layout
argument can be either torch.strided
or torch.sparse_coo
, for dense and sparse tensors respectively.details
argument offers a way to pass an arbitrary number of additional flags that customise and extend torchtyping
. Two flags are built-in by default. torchtyping.is_named
causes the names of tensor dimensions to be checked, and torchtyping.is_float
can be used to check that arbitrary floating point types are passed in. (Rather than just a specific one as with e.g. TensorType[torch.float32]
.) For discussion on how to customise torchtyping
with your own details
, see the further documentation.[]
. For example TensorType["batch": ..., "length", "channels", float, is_named]
.torchtyping.patch_typeguard()
torchtyping
integrates with typeguard
to perform runtime type checking. torchtyping.patch_typeguard()
should be called at the global level, and will patch typeguard
to check TensorType
s.
This function is safe to run multiple times. (It does nothing after the first run).
@typeguard.typechecked
, then torchtyping.patch_typeguard()
should be called any time before using @typeguard.typechecked
. For example you could call it at the start of each file using torchtyping
.typeguard.importhook.install_import_hook
, then torchtyping.patch_typeguard()
should be called any time before defining the functions you want checked. For example you could call torchtyping.patch_typeguard()
just once, at the same time as the typeguard
import hook. (The order of the hook and the patch doesn't matter.)typeguard
then torchtyping.patch_typeguard()
can be omitted altogether, and torchtyping
just used for documentation purposes.pytest --torchtyping-patch-typeguard
torchtyping
offers a pytest
plugin to automatically run torchtyping.patch_typeguard()
before your tests. pytest
will automatically discover the plugin, you just need to pass the --torchtyping-patch-typeguard
flag to enable it. Packages can then be passed to typeguard
as normal, either by using @typeguard.typechecked
, typeguard
's import hook, or the pytest
flag --typeguard-packages="your_package_here"
.
See the further documentation for:
flake8
and mypy
compatibility;torchtyping
;Author: patrick-kidger
Source Code: https://github.com/patrick-kidger/torchtyping
License: Apache-2.0 license
1653264240
Pyramid LDAP
pyramid_ldap
provides LDAP authentication services for your Pyramid application. Thanks to the ever-awesome SurveyMonkey for sponsoring the development of this package!
See the documentation at https://docs.pylonsproject.org/projects/pyramid_ldap/en/latest/ for more information.
This package will only work with Pyramid 1.3 and later.
pyramid_ldap
uses pyldap
which in turn requires libldap2
and libsasl2
development headers installed.
On Ubuntu 16.04 you can install them using the command apt-get install libldap2-dev libsasl2-dev
.
Author: Pylons
Source Code: https://github.com/Pylons/pyramid_ldap
License: View license
1653262860
Locust
Locust is an easy to use, scriptable and scalable performance testing tool. You define the behaviour of your users in regular Python code, instead of being constrained by a UI or domain specific language that only pretends to be real code. This makes Locust infinitely expandable and very developer friendly.
To get started right away, head over to the documentation.
If you want your users to loop, perform some conditional behaviour or do some calculations, you just use the regular programming constructs provided by Python. Locust runs every user inside its own greenlet (a lightweight process/coroutine). This enables you to write your tests like normal (blocking) Python code instead of having to use callbacks or some other mechanism. Because your scenarios are โjust pythonโ you can use your regular IDE, and version control your tests as regular code (as opposed to some other tools that use XML or binary formats)
from locust import HttpUser, task, between
class QuickstartUser(HttpUser):
wait_time = between(1, 2)
def on_start(self):
self.client.post("/login", json={"username":"foo", "password":"bar"})
@task
def hello_world(self):
self.client.get("/hello")
self.client.get("/world")
@task(3)
def view_item(self):
for item_id in range(10):
self.client.get(f"/item?id={item_id}", name="/item")
Locust makes it easy to run load tests distributed over multiple machines. It is event-based (using gevent), which makes it possible for a single process to handle many thousands concurrent users. While there may be other tools that are capable of doing more requests per second on a given hardware, the low overhead of each Locust user makes it very suitable for testing highly concurrent workloads.
Locust has a user friendly web interface that shows the progress of your test in real-time. You can even change the load while the test is running. It can also be run without the UI, making it easy to use for CI/CD testing.
Even though Locust primarily works with web sites/services, it can be used to test almost any system or protocol. Just write a client for what you want to test, or explore some created by the community.
Locust's code base is intentionally kept small and doesn't solve everything out of the box. Instead, we try to make it easy to adapt to any situation you may come across, using regular Python code. If you want to send reporting data to that database & graphing system you like, wrap calls to a REST API to handle the particulars of your system or run a totally custom load pattern, there is nothing stopping you!
Open source licensed under the MIT license (see LICENSE file for details).
Author: locustio
Source Code: https://github.com/locustio/locust
License: MIT License
1653261060
Python pandas tutorial for beginners on how perform addition, substraction on two different series or dataframe on their numerical values.
1653255480
tox automation project
Command line driven CI frontend and development task automation tool
At its core tox provides a convenient way to run arbitrary commands in isolated environments to serve as a single entry point for build, test and release activities.
tox is highly configurable and pluggable.
tox is mainly used as a command line tool and needs a tox.ini
or a tool.tox
section in pyproject.toml
containing the configuration.
To test a simple project that has some tests, here is an example with a tox.ini
in the root of the project:
[tox]
envlist = py37,py38
[testenv]
deps = pytest
commands = pytest
$ tox
[lots of output from what tox does]
[lots of output from commands that were run]
__________________ summary _________________
py37: commands succeeded
py38: commands succeeded
congratulations :)
tox created two testenvs
- one based on Python 3.7 and one based on Python 3.8, it installed pytest in them and ran the tests. The report at the end summarizes which testenvs
have failed and which have succeeded.
Note: To learn more about what you can do with tox, have a look at the collection of examples in the documentation or existing projects using tox.
tox creates virtual environments for all configured so-called testenvs
, it then installs the project and other necessary dependencies and runs the configured set of commands. See system overview for more details.
Documentation for tox can be found at Read The Docs.
For the fastest and interactive feedback please join our server. If you have questions or suggestions you can first check if they have already been answered or discussed on our issue tracker. On Stack Overflow (tagged with tox
).
Contributions are welcome. See contributing and our Contributor Covenant Code of Conduct.
Currently, the code and the issues are hosted on GitHub.
The project is licensed under MIT.
Author: tox-dev
Source Code: https://github.com/tox-dev/tox
License: MIT License
1653253740
Python tutorial for beginners on how to remove duplicate values from python pandas dataframe.
I have first shown the duplicated function of pandas which returns boolean value whether a particular row is duplicate or not.
After that I have sum method that you can chain with duplicated method to return the count' of rows that are duplicated in a dataset
After that I have shown how to find whether a particular column is having duplicate values or not as well as getting the count of duplicated values of a column.
By default duplicated pandas function keep the first row or value while identifying the duplicate value however I have shown you how can you keep the last value as an out of unique value dataset.
Finally I have shown the function to drop duplicate values out from the dataset and its various parameters like subset or keep etc. to further define the output of the unique values.
1653251160
Mypy plugin and stubs for SQLAlchemy
This package contains type stubs and a mypy plugin to provide more precise static types and type inference for SQLAlchemy framework. SQLAlchemy uses some Python "magic" that makes having precise types for some code patterns problematic. This is why we need to accompany the stubs with mypy plugins. The final goal is to be able to get precise types for most common patterns. Currently, basic operations with models are supported. A simple example:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
user = User(id=42, name=42) # Error: Incompatible type for "name" of "User"
# (got "int", expected "Optional[str]")
user.id # Inferred type is "int"
User.name # Inferred type is "Column[Optional[str]]"
Some auto-generated attributes are added to models. Simple relationships are supported but require models to be imported:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from models.address import Address
...
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
address = relationship('Address') # OK, mypy understands string references.
The next step is to support precise types for table definitions (e.g. inferring Column[Optional[str]]
for users.c.name
, currently it is just Column[Any]
), and precise types for results of queries made using query()
and select()
.
Install latest published version as:
pip install -U sqlalchemy-stubs
Important: you need to enable the plugin in your mypy config file:
[mypy]
plugins = sqlmypy
To install the development version of the package:
git clone https://github.com/dropbox/sqlalchemy-stubs
cd sqlalchemy-stubs
pip install -U .
First, clone the repo and cd into it, like in Installation, then:
git submodule update --init --recursive
pip install -r dev-requirements.txt
Then, to run the tests, simply:
pytest
The package is currently in alpha stage. See issue tracker for bugs and missing features. If you want to contribute, a good place to start is help-wanted
label.
Currently, some basic use cases like inferring model field types are supported. The long term goal is to be able to infer types for more complex situations like correctly inferring columns in most compound queries.
External contributions to the project should be subject to Dropbox Contributor License Agreement (CLA).
Author: dropbox
Source Code: https://github.com/dropbox/sqlalchemy-stubs
License: Apache-2.0 license