Royce Reinger

Royce Reinger

1676482920

GENDIS: GENetic DIscovery of Shapelets

GENDIS

GENetic DIscovery of Shapelets

In the time series classification domain, shapelets are small subseries that are discriminative for a certain class. It has been shown that by projecting the original dataset to a distance space, where each axis corresponds to the distance to a certain shapelet, classifiers are able to achieve state-of-the-art results on a plethora of datasets.

This repository contains an implementation of GENDIS, an algorithm that searches for a set of shapelets in a genetic fashion. The algorithm is insensitive to its parameters (such as population size, crossover and mutation probability, ...) and can quickly extract a small set of shapelets that is able to achieve predictive performances similar (or better) to that of other shapelet techniques.

evolving_shaps.gif

Installation

We currently support Python 3.5 & Python 3.6. For installation, there are two alternatives:

  1. Clone the repository https://github.com/IBCNServices/GENDIS.git and run (python3 -m) pip -r install requirements.txt
  2. GENDIS is hosted on PyPi. You can just run (python3 -m) pip install gendis to add gendis to your dist-packages (you can use it from everywhere).

Make sure NumPy and Cython is already installed (pip install numpy and pip install Cython), since that is required for the setup script.

Tutorial & Example

1. Loading & preprocessing the datasets

In a first step, we need to construct at least a matrix with timeseries (X_train) and a vector with labels (y_train). Additionally, test data can be loaded as well in order to evaluate the pipeline in the end.

import pandas as pd
# Read in the datafiles
train_df = pd.read_csv(<DATA_FILE>)
test_df = pd.read_csv(<DATA_FILE>)
# Split into feature matrices and label vectors
X_train = train_df.drop('target', axis=1)
y_train = train_df['target']
X_test = test_df.drop('target', axis=1)
y_test = test_df['target']

2. Creating a GeneticExtractor object

Construct the object. For a list of all possible parameters, and a description, please refer to the documentation in the code

from gendis.genetic import GeneticExtractor
genetic_extractor = GeneticExtractor(population_size=50, iterations=25, verbose=True, 
                                     mutation_prob=0.3, crossover_prob=0.3, 
                                     wait=10, max_len=len(X_train) // 2)

3. Fit the GeneticExtractor and construct distance matrix

shapelets = genetic_extractor.fit(X_train, y_train)
distances_train = genetic_extractor.transform(X_train)
distances_test = genetic_extractor.transform(X_test)

4. Fit ML classifier on constructed distance matrix

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
lr = LogisticRegression()
lr.fit(distances_train, y_train)

print('Accuracy = {}'.format(accuracy_score(y_test, lr.predict(distances_test))))

Example notebook

A simple example is provided in this notebook

Data

All datasets in this repository are downloaded from timeseriesclassification. Please refer to them appropriately when using any dataset.

Paper experiments

In order to reproduce the results from the corresponding paper, please check out this directory.

Tests

We provide a few doctests and unit tests. To run the doctests: python3 -m doctest -v <FILE>, where <FILE> is the Python file you want to run the doctests from. To run unit tests: nose2 -v

Contributing, Citing and Contact

If you have any questions, are experiencing bugs in the GENDIS implementation, or would like to contribute, please feel free to create an issue/pull request in this repository or take contact with me at gilles(dot)vandewiele(at)ugent(dot)be

If you use GENDIS in your work, please use the following citation:

@article{vandewiele2021gendis,
  title={GENDIS: Genetic Discovery of Shapelets},
  author={Vandewiele, Gilles and Ongenae, Femke and Turck, Filip De},
  journal={Sensors},
  volume={21},
  number={4},
  pages={1059},
  year={2021},
  publisher={Multidisciplinary Digital Publishing Institute}
}

Download Details:

Author: IBCNServices
Source Code: https://github.com/IBCNServices/GENDIS 
License: View license

#machinelearning #python #datamining #algorithms 

What is GEEK

Buddha Community

GENDIS: GENetic DIscovery of Shapelets
Royce Reinger

Royce Reinger

1676482920

GENDIS: GENetic DIscovery of Shapelets

GENDIS

GENetic DIscovery of Shapelets

In the time series classification domain, shapelets are small subseries that are discriminative for a certain class. It has been shown that by projecting the original dataset to a distance space, where each axis corresponds to the distance to a certain shapelet, classifiers are able to achieve state-of-the-art results on a plethora of datasets.

This repository contains an implementation of GENDIS, an algorithm that searches for a set of shapelets in a genetic fashion. The algorithm is insensitive to its parameters (such as population size, crossover and mutation probability, ...) and can quickly extract a small set of shapelets that is able to achieve predictive performances similar (or better) to that of other shapelet techniques.

evolving_shaps.gif

Installation

We currently support Python 3.5 & Python 3.6. For installation, there are two alternatives:

  1. Clone the repository https://github.com/IBCNServices/GENDIS.git and run (python3 -m) pip -r install requirements.txt
  2. GENDIS is hosted on PyPi. You can just run (python3 -m) pip install gendis to add gendis to your dist-packages (you can use it from everywhere).

Make sure NumPy and Cython is already installed (pip install numpy and pip install Cython), since that is required for the setup script.

Tutorial & Example

1. Loading & preprocessing the datasets

In a first step, we need to construct at least a matrix with timeseries (X_train) and a vector with labels (y_train). Additionally, test data can be loaded as well in order to evaluate the pipeline in the end.

import pandas as pd
# Read in the datafiles
train_df = pd.read_csv(<DATA_FILE>)
test_df = pd.read_csv(<DATA_FILE>)
# Split into feature matrices and label vectors
X_train = train_df.drop('target', axis=1)
y_train = train_df['target']
X_test = test_df.drop('target', axis=1)
y_test = test_df['target']

2. Creating a GeneticExtractor object

Construct the object. For a list of all possible parameters, and a description, please refer to the documentation in the code

from gendis.genetic import GeneticExtractor
genetic_extractor = GeneticExtractor(population_size=50, iterations=25, verbose=True, 
                                     mutation_prob=0.3, crossover_prob=0.3, 
                                     wait=10, max_len=len(X_train) // 2)

3. Fit the GeneticExtractor and construct distance matrix

shapelets = genetic_extractor.fit(X_train, y_train)
distances_train = genetic_extractor.transform(X_train)
distances_test = genetic_extractor.transform(X_test)

4. Fit ML classifier on constructed distance matrix

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
lr = LogisticRegression()
lr.fit(distances_train, y_train)

print('Accuracy = {}'.format(accuracy_score(y_test, lr.predict(distances_test))))

Example notebook

A simple example is provided in this notebook

Data

All datasets in this repository are downloaded from timeseriesclassification. Please refer to them appropriately when using any dataset.

Paper experiments

In order to reproduce the results from the corresponding paper, please check out this directory.

Tests

We provide a few doctests and unit tests. To run the doctests: python3 -m doctest -v <FILE>, where <FILE> is the Python file you want to run the doctests from. To run unit tests: nose2 -v

Contributing, Citing and Contact

If you have any questions, are experiencing bugs in the GENDIS implementation, or would like to contribute, please feel free to create an issue/pull request in this repository or take contact with me at gilles(dot)vandewiele(at)ugent(dot)be

If you use GENDIS in your work, please use the following citation:

@article{vandewiele2021gendis,
  title={GENDIS: Genetic Discovery of Shapelets},
  author={Vandewiele, Gilles and Ongenae, Femke and Turck, Filip De},
  journal={Sensors},
  volume={21},
  number={4},
  pages={1059},
  year={2021},
  publisher={Multidisciplinary Digital Publishing Institute}
}

Download Details:

Author: IBCNServices
Source Code: https://github.com/IBCNServices/GENDIS 
License: View license

#machinelearning #python #datamining #algorithms 

Myriam Rogahn

Myriam Rogahn

1598775060

Optimization Techniques: Genetic Algorithm

In complex machine learning models, the performance usually depends on multiple input parameters. In order to get the optimal model, the parameters must be properly tuned. However, when there are multiple parameter variables, each ranging across a wide spectrum of values, there are too many possible configurations for each set of parameters to be tested. In these cases, optimization methods should be used to attain the optimal input parameters without spending vast amounts of time finding them.

Image for post

In the diagram above, it shows the distribution of the model based on only two parameters. As evident in the example shown, it is not always an easy task to find the maximum or minimum of the curve. This is why optimization methods and algorithms are crucial in the field of machine learning.

Genetic Algorithm

The most commonly used optimization strategy are Genetic Algorithms. Genetic Algorithms are based off of Darwin’s theory of natural selection. It is relatively easy to implement and there is a lot of flexibility for the setup of the algorithm so that it can be applied to a wide range of problems.

Choosing a Fitness Function

To start off, there must be a fitness function that measures how well a set of input parameters perform. Solutions with a higher fitness derived from a fitness function will be better than ones with a lower fitness.

For example, if a solution has a cost of x + y + z, then the fitness function should try to minimize the cost. This can be done with the following fitness function

Image for post

#genetic-algorithm #optimization #genetics #optimization-algorithms #machine-learning

Beth Nabimanya

Beth Nabimanya

1624882500

A Guide to Genetic ‘Learning’ Algorithms for Optimization

In a broader mathematical or computational perspective, an optimization problem is defined as a problem of finding the best solution from all feasible solutions. In terms of Machine Learning and Artificial Intelligence, two significant algorithms that perform these tasks are Reinforcement Learning and Genetic Algorithms. They serve the purpose of finding the ‘best fit’ solutions from a range of possible solutions for a given problem statement. In the article that follows below, we will be working closely on these algorithms and will see their implementation in action on an Image Processing problem.

What is a Genetic Algorithm (GA)?

Genetic algorithms are random, adaptive heuristic search algorithms that act on a population of doable solutions. they need loosely supported the mechanics of population biology and choice.

Genetic algorithms are based on the ideas of natural selection and genetics. New solutions are typically made by ‘mutating’ members of this population, and by ‘mating’ 2 resolutions along to create a replacement solution.

The upper solutions are selected to breed and change and so the more severe ones are discarded. They are probabilistic search methods; this implies that the states that they explore are not determined entirely by the properties of the problems. A random method helps to guide the search. Genetic algorithms are utilized in AI like different search algorithms utilized in AI — to seem for potential solutions to hunt out one that solves the matter.

#artificial-intelligence #genetics #algorithms #data-science #genetic ‘learning’ algorithms

Andre Fisher

Andre Fisher

1667894340

Awesome Asset Discovery: List of Awesome Asset Discovery Resources

Awesome Asset Discovery

Asset Discovery is the initial phase of any security assessment engagement, be it offensive or defensive. With the evolution of information technology, the scope and definition of assets has also evolved.

Earlier the servers, workstations and websites were primary IT assets of an organization, but today this definition is very limiting and should include anything and everything an organization and its entities has their data on (knowingly or unknowingly). The scope of ownership could differ, but it does not limit the attack surface, for example if an organization puts out open source code on Github, they are not the owner of Github but of the data they put under their repositories. In a scenario where some organization secret has been put on this Github account, it could pose a threat equal or more than running a vulnerable service.

We have explored this aspect of assets in our blog post here.

Through this repository, we want to put out a list of curated resources which help during asset discovery phase of a security assessment engagement. We welcome suggestions and contributions from the community in terms of resources as well as categories.

To know more about our Attack Surface Management platform, check out NVADR.

↑Content Discovery

  • rustbuster: Files, directories and vhost buster written in Rust.

↑IP Address Discovery

  • Mxtoolbox: Bulk Domain/IP lookup tool
  • Domaintoipconverter: Bulk domain to IP converter
  • Massdns: A DNS resolver utility for bulk lookups
  • Googleapps Dig: Online Dig tool by Google
  • DataSploit (IP Address Modules): An OSINT Framework to perform various recon techniques
  • Domain Dossier: Investigate domains and IP addresses
  • Bgpview: Search ASN, IPv4/IPv6 or resource name
  • Hurricane Electric BGP Toolkit: Keyword to ASN lookup
  • Viewdns: Multiple domain/IP tools
  • Ultratools ipv6Info: Multiple information related to IPv6 address
  • Whois: Command line utility usually used to find information about registered users/assignees of an Internet resource.
  • ICANN Whois: Whois service by Internet Corporation for Assigned Names and Numbers (ICANN)
  • Nslookup Linux / Windows: Command line utility usually used for querying the DNS records
  • bgp : Internet Backbone and Colocation Provider ... Hurricane Electric IP Transit. Our Global Internet Backbone provides IP Transit with low latency, access to thousands of networks, and dual-stack

↑Domain / Subdomain Discovery

  • SubFinder: SubFinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.
  • Amass: A subdomain enumeration utility
  • Sublist3r: Subdomains enumeration tool with multiple sources
  • Aiodnsbrute: Asynchronous DNS brute force utility
  • LDNS: A DNS library useful for DNS tool programming
  • Dns-nsec3-enum: Nmap NSE Script for NSEC3 walking
  • Nsec3map: A tool to NSEC and NSEC3 walking
  • Crt.sh: Domain certificate Search
  • Ct-exposer: A tool to discovers sub-domains by searching Certificate Transparency logs
  • Certgraph: A tool to crawl the graph of certificate Alternate Names
  • Appsecco - The art of subdomain enumeration: The supplement material for the book "The art of sub-domain enumeration"
  • SSLScrape: A scanning tool to scrape hostnames from SSL certificates
  • Wolframalpha: Computational knowledge engine
  • Project Sonar: Forward DNS Data
  • Project Sonar: Reverse DNS Data
  • GoBuster: Directory/File, DNS and VHost busting tool written in Go
  • Bluto: Recon, Subdomain Bruting, Zone Transfers

↑Email Discovery

  • Hunter: Email search for a domain
  • Skrapp: Browser addon to find emails on Linkedin
  • Email Extractor: Chrome extension to extract emails from web pages
  • Convertcsv: Online tool to extract email addresses in text, web pages, data files etc.
  • linkedin2username: OSINT Tool: Generate username lists for companies on LinkedIn
  • Office365UserEnum: Enumerate valid usernames from Office 365 using ActiveSync.

↑Network / Port Scanning

  • Zmap: A fast network scanner designed for Internet-wide network surveys
  • Masscan: An asynchronously TCP port scanner
  • ZMapv6: A modified version of Zmap with IPv6 support.
  • Nmap: A free and open source utility for network discovery. The most popular port scanner.

↑Business Communication Infrastructure Discovery

  • Mxtoolbox: Online tool to check mail exchanger (MX) records
  • MicroBurst: PowerShell based Azure security assessment scripts
  • Lyncsmash: Tools to enumerate and attack self-hosted Lync/Skype for Business
  • Enumeration-as-a-Service: Script for SaaS offering enumeration through DNS queries
  • ruler : A tool to abuse Exchange services

↑Source Code Aggregators / Search - Information Discovery

  • Github: Github Advanced Search
  • Bitbucket: Bitbucket Search using Google
  • Gitrob: Reconnaissance tool for GitHub organizations
  • Gitlab: Search Gitlab projects
  • Publicwww: Source Code Search Engine
  • builtwith : Web technology information profiler tool. Find out what a website is built with.

↑Cloud Infrastructure Discovery

↑Company Information and Associations

↑Internet Survey Data

  • Project Resonance: RedHunt Labs’s Internet wide surveys to study and understand the security state of the Internet.
  • Project Sonar: Rapid7’s internet-wide surveys data across different services and protocols
  • Scans.io: Internet-Wide Scan Data Repository, hosted by the ZMap Team
  • Portradar: Free and open port scan data by packet.tel

↑Social Media / Employee Profiling

↑Data Leaks

  • Dumpmon: A twitter bot which monitors multiple paste sites for password dumps and other sensitive information
  • Pastebin_scraper: Automated tool to monitor pastebin for interesting information
  • Scavenger: Paste sites crawler (bot) looking for leaked credentials
  • Pwnbin: Python based Pastebin crawler for keywords.
  • PwnedOrNot: Tool to find passwords for compromised accounts

↑Internet Scan / Archived Information

  • Cachedviews: Cached view of pages on the Internet from multiple sources
  • Wayback Machine: Internet Archive
  • Shodan: Search engine for Internet-connected devices
  • Censys: Another search engine for internet-connected devices
  • Zoomeye: Cyberspace Search Engine

Contributing

In case you would like to add information to this repository or suggest some ideas, please use one of the following options:

Download Details:

Author: redhuntlabs
Source Code: https://github.com/redhuntlabs/Awesome-Asset-Discovery

License: CC0-1.0 license

#Discovery 

Linda nano

Linda nano

1624323600

XRP Ripple News - SEC Motion to DISMISS Ripple XRP Defense & Major Discovery Battle

📺 The video in this post was made by Crypto 2103
The origin of the article: https://www.youtube.com/watch?v=bzCxsLKp79A
🔺 DISCLAIMER: The article is for information sharing. The content of this video is solely the opinions of the speaker who is not a licensed financial advisor or registered investment advisor. Not investment advice or legal advice.
Cryptocurrency trading is VERY risky. Make sure you understand these risks and that you are responsible for what you do with your money
🔥 If you’re a beginner. I believe the article below will be useful to you ☞ What You Should Know Before Investing in Cryptocurrency - For Beginner
⭐ ⭐ ⭐The project is of interest to the community. Join to Get free ‘GEEK coin’ (GEEKCASH coin)!
☞ **-----CLICK HERE-----**⭐ ⭐ ⭐
Thanks for visiting and watching! Please don’t forget to leave a like, comment and share!

#bitcoin #blockchain #xrp ripple news #xrp #xrp ripple news - sec motion to dismiss ripple xrp defense & major discovery battle #xrp ripple news - sec motion to dismiss ripple xrp defense & major discovery battle