Python RegEx (Regular Expressions) Tutorial with Examples
Regular Expressions in Python | Set 1
Module Regular Expressions(RE) specifies a set of strings(pattern) that matches it.
To understand the RE analogy, MetaCharacters are useful, important and will be used in functions of module re.
There are a total of 14 metacharacters and will be discussed as they follow into functions:
\ Used to drop the special meaning of character
following it (discussed below)
[] Represent a character class
^ Matches the beginning
$ Matches the end
. Matches any character except newline
? Matches zero or one occurrence.
| Means OR (Matches with any of the characters
separated by it.
* Any number of occurrences (including 0 occurrences)
+ One or more occurrences
{} Indicate number of occurrences of a preceding RE
to match.
() Enclose a group of REs
- Function compile()
Regular expressions are compiled into pattern objects, which have methods for various operations such as searching for pattern matches or performing string substitutions.
# Module Regular Expression is imported using __import__().
import re
# compile() creates regular expression character class [a-e],
# which is equivalent to [abcde].
# class [abcde] will match with string with 'a', 'b', 'c', 'd', 'e'.
p = re.compile('[a-e]')
# findall() searches for the Regular Expression and return a list upon finding
print(p.findall("Aye, said Mr. Gibenson Stark"))
Output:
['e', 'a', 'd', 'b', 'e', 'a']
Understanding the Output:
First occurrence is ‘e’ in “Aye” and not ‘A’, as it being Case Sensitive.
Next Occurrence is ‘a’ in “said”, then ‘d’ in “said”, followed by ‘b’ and ‘e’ in “Gibenson”, the Last ‘a’ matches with “Stark”.
Metacharacter backslash ‘\’ has a very important role as it signals various sequences. If the backslash is to be used without its special meaning as metacharacter, use’\’
\d Matches any decimal digit, this is equivalent
to the set class [0-9].
\D Matches any non-digit character.
\s Matches any whitespace character.
\S Matches any non-whitespace character
\w Matches any alphanumeric character, this is
equivalent to the class [a-zA-Z0-9_].
\W Matches any non-alphanumeric character.
Set class [\s,.] will match any whitespace character, ‘,’, or,’.’ .
import re
# \d is equivalent to [0-9].
p = re.compile('\d')
print(p.findall("I went to him at 11 A.M. on 4th July 1886"))
# \d+ will match a group on [0-9], group of one or greater size
p = re.compile('\d+')
print(p.findall("I went to him at 11 A.M. on 4th July 1886"))
Output:
['1', '1', '4', '1', '8', '8', '6']
['11', '4', '1886']
import re
# \w is equivalent to [a-zA-Z0-9_].
p = re.compile('\w')
print(p.findall("He said * in some_lang."))
# \w+ matches to group of alphanumeric character.
p = re.compile('\w+')
print(p.findall("I went to him at 11 A.M., he said *** in some_language."))
# \W matches to non alphanumeric characters.
p = re.compile('\W')
print(p.findall("he said *** in some_language."))
Output:
['1', '1', '4', '1', '8', '8', '6']
['11', '4', '1886']
import re
# \w is equivalent to [a-zA-Z0-9_].
p = re.compile('\w')
print(p.findall("He said * in some_lang."))
# \w+ matches to group of alphanumeric character.
p = re.compile('\w+')
print(p.findall("I went to him at 11 A.M., he said *** in some_language."))
# \W matches to non alphanumeric characters.
p = re.compile('\W')
print(p.findall("he said *** in some_language."))
Output:
['H', 'e', 's', 'a', 'i', 'd', 'i', 'n', 's', 'o', 'm', 'e', '_', 'l', 'a', 'n', 'g']
['I', 'went', 'to', 'him', 'at', '11', 'A', 'M', 'he', 'said', 'in', 'some_language']
[' ', ' ', '*', '*', '*', ' ', ' ', '.']
import re
# '*' replaces the no. of occurrence of a character.
p = re.compile('ab*')
print(p.findall("ababbaabbb"))
Output:
['ab', 'abb', 'a', 'abbb']
Understanding the Output:
Our RE is ab*, which ‘a’ accompanied by any no. of ‘b’s, starting from 0.
Output ‘ab’, is valid because of singe ‘a’ accompanied by single ‘b’.
Output ‘abb’, is valid because of singe ‘a’ accompanied by 2 ‘b’.
Output ‘a’, is valid because of singe ‘a’ accompanied by 0 ‘b’.
Output ‘abbb’, is valid because of singe ‘a’ accompanied by 3 ‘b’.
- Function split()
Split string by the occurrences of a character or a pattern, upon finding that pattern, the remaining characters from the string are returned as part of the resulting list.
Syntax :
re.split(pattern, string, maxsplit=0, flags=0)
The First parameter, pattern denotes the regular expression, string is the given string in which pattern will be searched for and in which splitting occurs, maxsplit if not provided is considered to be zero ‘0’, and if any nonzero value is provided, then at most that many splits occurs. If maxsplit = 1, then the string will split once only, resulting in a list of length 2. The flags are very useful and can help to shorten code, they are not necessary parameters, eg: flags = re.IGNORECASE, In this split, case will be ignored.
from re import split
# '\W+' denotes Non-Alphanumeric Characters or group of characters
# Upon finding ',' or whitespace ' ', the split(), splits the string from that point
print(split('\W+', 'Words, words , Words'))
print(split('\W+', "Word's words Words"))
# Here ':', ' ' ,',' are not AlphaNumeric thus, the point where splitting occurs
print(split('\W+', 'On 12th Jan 2016, at 11:02 AM'))
# '\d+' denotes Numeric Characters or group of characters
# Splitting occurs at '12', '2016', '11', '02' only
print(split('\d+', 'On 12th Jan 2016, at 11:02 AM'))
Output:
['Words', 'words', 'Words']
['Word', 's', 'words', 'Words']
['On', '12th', 'Jan', '2016', 'at', '11', '02', 'AM']
['On ', 'th Jan ', ', at ', ':', ' AM']
import re
# Splitting will occurs only once, at '12', returned list will have length 2
print(re.split('\d+', 'On 12th Jan 2016, at 11:02 AM', 1))
# 'Boy' and 'boy' will be treated same when flags = re.IGNORECASE
print(re.split('[a-f]+', 'Aey, Boy oh boy, come here', flags = re.IGNORECASE))
print(re.split('[a-f]+', 'Aey, Boy oh boy, come here'))
Output:
['On ', 'th Jan 2016, at 11:02 AM']
['', 'y, ', 'oy oh ', 'oy, ', 'om', ' h', 'r', '']
['A', 'y, Boy oh ', 'oy, ', 'om', ' h', 'r', '']
- Function sub()
Syntax:
re.sub(pattern, repl, string, count=0, flags=0)
The ‘sub’ in the function stands for SubString, a certain regular expression pattern is searched in the given string(3rd parameter), and upon finding the substring pattern is replaced by repl(2nd parameter), count checks and maintains the number of times this occurs.
import re
# Regular Expression pattern 'ub' matches the string at "Subject" and "Uber".
# As the CASE has been ignored, using Flag, 'ub' should match twice with the string
# Upon matching, 'ub' is replaced by '~*' in "Subject", and in "Uber", 'Ub' is replaced.
print(re.sub('ub', '~*' , 'Subject has Uber booked already', flags = re.IGNORECASE))
# Consider the Case Sensitivity, 'Ub' in "Uber", will not be reaplced.
print(re.sub('ub', '~*' , 'Subject has Uber booked already'))
# As count has been given value 1, the maximum times replacement occurs is 1
print(re.sub('ub', '~*' , 'Subject has Uber booked already', count=1, flags = re.IGNORECASE))
# 'r' before the patter denotes RE, \s is for start and end of a String.
print(re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam', flags=re.IGNORECASE))
Output
S~*ject has ~*er booked already
S~*ject has Uber booked already
S~*ject has Uber booked already
Baked Beans & Spam
- Function subn()
Syntax:
re.subn(pattern, repl, string, count=0, flags=0)
subn() is similar to sub() in all ways, except in its way to providing output. It returns a tuple with count of total of replacement and the new string rather than just the string.
import re
print(re.subn('ub', '~*' , 'Subject has Uber booked already'))
t = re.subn('ub', '~*' , 'Subject has Uber booked already', flags = re.IGNORECASE)
print(t)
print(len(t))
# This will give same output as sub() would have
print(t[0])
Output
('S~*ject has Uber booked already', 1)
('S~*ject has ~*er booked already', 2)
Length of Tuple is: 2
S~*ject has ~*er booked already
- Function escape()
Syntax:
re.escape(string)
Return string with all non-alphanumerics backslashed, this is useful if you want to match an arbitrary literal string that may have regular expression metacharacters in it.
import re
# escape() returns a string with BackSlash '\', before every Non-Alphanumeric Character
# In 1st case only ' ', is not alphanumeric
# In 2nd case, ' ', caret '^', '-', '[]', '\' are not alphanumeric
print(re.escape("This is Awseome even 1 AM"))
print(re.escape("I Asked what is this [a-9], he said \t ^WoW"))
Output
This\ is\ Awseome\ even\ 1\ AM
I\ Asked\ what\ is\ this\ \[a\-9\]\,\ he\ said\ \ \ \^WoW
Regular Expressions in Python | Set 2 (Search, Match and Find All)
The module re provides support for regular expressions in Python. Below are main methods in this module.
Searching an occurrence of pattern
re.search() : This method either returns None (if the pattern doesn’t match), or a re.MatchObject that contains information about the matching part of the string. This method stops after the first match, so this is best suited for testing a regular expression more than extracting data.
# A Python program to demonstrate working of re.match().
import re
# Lets use a regular expression to match a date string
# in the form of Month name followed by day number
regex = r"([a-zA-Z]+) (\d+)"
match = re.search(regex, "I was born on June 24")
if match != None:
# We reach here when the expression "([a-zA-Z]+) (\d+)"
# matches the date string.
# This will print [14, 21), since it matches at index 14
# and ends at 21.
print "Match at index %s, %s" % (match.start(), match.end())
# We us group() method to get all the matches and
# captured groups. The groups contain the matched values.
# In particular:
# match.group(0) always returns the fully matched string
# match.group(1) match.group(2), ... return the capture
# groups in order from left to right in the input string
# match.group() is equivalent to match.group(0)
# So this will print "June 24"
print "Full match: %s" % (match.group(0))
# So this will print "June"
print "Month: %s" % (match.group(1))
# So this will print "24"
print "Day: %s" % (match.group(2))
else:
print "The regex pattern does not match."
Output :
Match at index 14, 21
Full match: June 24
Month: June
Day: 24
Matching a Pattern with Text
re.match() : This function attempts to match pattern to whole string. The re.match function returns a match object on success, None on failure.
re.match(pattern, string, flags=0)
pattern : Regular expression to be matched.
string : String where p attern is searched
flags : We can specify different flags
using bitwise OR (|).
# A Python program to demonstrate working
# of re.match().
import re
# a sample function that uses regular expressions
# to find month and day of a date.
def findMonthAndDate(string):
regex = r"([a-zA-Z]+) (\d+)"
match = re.match(regex, string)
if match == None:
print "Not a valid date"
return
print "Given Data: %s" % (match.group())
print "Month: %s" % (match.group(1))
print "Day: %s" % (match.group(2))
# Driver Code
findMonthAndDate("Jun 24")
print("")
findMonthAndDate("I was born on June 24")
Finding all occurrences of a pattern
re.findall() : Return all non-overlapping matches of pattern in string, as a list of strings. The string is scanned left-to-right, and matches are returned in the order found (Source : Python Docs).
# A Python program to demonstrate working of
# findall()
import re
# A sample text string where regular expression
# is searched.
string = """Hello my Number is 123456789 and
my friend's number is 987654321"""
# A sample regular expression to find digits.
regex = '\d+'
match = re.findall(regex, string)
print(match)
# This example is contributed by Ayush Saluja.
Output :
['123456789', '987654321']
Regular expression is a vast topic. It’s a complete library. Regular expressions can do a lot of stuff. You can Match, Search, Replace, Extract a lot of data. For example, below small code is so powerful that it can extract email address from a text. So we can make our own Web Crawlers and scrappers in python with easy.Look at below regex.
# extract all email addresses and add them into the resulting set
new_emails = set(re.findall(r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+",
text, re.I))
We will soon be discussing more methods on regular expressions.
Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above
#python #webdev #machinelearning #datascience
