5 Favorite PHP Libraries for File Manipulation and MIME Type Detection

In today's post we will learn about 5 Favorite PHP Libraries for File Manipulation and MIME Type Detection.

What is MIME type detection?

The most simple way to detect the MIME type of any file is to use MIME_Type's static autoDetect() method. It will try to determine the file's type and return it as a string. If an error occurs, a PEAR_Error object is returned. By default, only the plain MIME type will be returned, without any comments or parameters.

Table of contents:

  • CSV - A CSV data manipulation library.
  • Flysystem - Abstraction for local and remote filesystems.
  • Gaufrette - A filesystem abstraction layer.
  • PHP FFmpeg - A wrapper for the FFmpeg video library.
  • UnifiedArchive - A unified reader and writer of compressed archives.

1 - CSV:

A CSV data manipulation library.

Csv is a library to ease parsing, writing and filtering CSV in PHP. The library goal is to be powerful while remaining lightweight, by utilizing PHP native classes whenever possible.


  • Easy to use API
  • Read and Write to CSV documents in a memory efficient and scalable way
  • Support PHP stream filtering capabilities
  • Transform CSV documents into popular format (JSON, XML or HTML)
  • Fully documented
  • Fully unit tested
  • Framework-agnostic

System Requirements

You need the mbstring extension to use Csv and the latest stable version of PHP is recommended.

Please find below the PHP support for Csv version 9.

Min. Library VersionMin. PHP VersionMax. Supported PHP Version
9.0.0PHP 7.0.10PHP 7.1.x
9.1.2PHP 7.0.10PHP 7.2.x
9.2.0PHP 7.0.10PHP 7.4.x
9.6.0PHP 7.2.5PHP 7.4.x
9.6.2PHP 7.2.5PHP 8.0.x
9.7.0PHP 7.3.0PHP 8.0.x
9.7.3PHP 7.3.0PHP 8.1.x
9.8.0PHP 7.4.0PHP 8.1.x


Install Csv using Composer.

composer require league/csv


Warning: If your CSV document was created or is read on a Macintosh computer, add the following lines before using the library to help PHP detect line ending.

if (!ini_get("auto_detect_line_endings")) {
    ini_set("auto_detect_line_endings", '1');


The library has a :

To run the tests, run the following command from the project folder.

composer test

View on Github

2 - Flysystem:

Abstraction for local and remote filesystems.

Flysystem is a file storage library for PHP. It provides one interface to interact with many types of filesystems. When you use Flysystem, you're not only protected from vendor lock-in, you'll also have a consistent experience for which ever storage is right for you.

Getting Started

Officially supported adapters

Third party Adapters

You can always create an adapter yourself.


If you discover any security related issues, please email info@frankdejonge.nl instead of using the issue tracker.

View on Github

3 - Gaufrette:

A filesystem abstraction layer.

Gaufrette provides a filesystem abstraction layer.

Why use Gaufrette?

Imagine you have to manage a lot of medias in a PHP project. Lets see how to take this situation in your advantage using Gaufrette.

The filesystem abstraction layer permits you to develop your application without the need to know where all those medias will be stored and how.

Another advantage of this is the possibility to update the files location without any impact on the code apart from the definition of your filesystem. In example, if your project grows up very fast and if your server reaches its limits, you can easily move your medias in an Amazon S3 server or any other solution.


Requires :

  • docker-ce
  • docker-compose

Create .env file :

$ make docker.dev

and configure it as you want.

Build the php docker image :

$ make docker.build

Install dependencies :

$ make docker.all-deps

Run tests :

$ make docker.tests

You can also use a different php version, simply set the PHP_VERSION env var to any of these values when calling a make target :

See the docker-compose.yml file for more details.

You'll need to clear the previously installed dependencies when switching from a version to an other, to do so, run :

$ make clear-deps
$ PHP_VERSION=<the_version_you_want_to_use> make build install-deps

Apply Coding Standards

You should check for CS violations by using

$ make php-cs-compare

and fix them with

$ make php-cs-fix


This project does not have any stable release yet but we do not want to break BC now.

View on Github

4 - PHP FFmpeg:

A wrapper for the FFmpeg video library.

An Object-Oriented library to convert video/audio files with FFmpeg / AVConv.


This library requires PHP 8.0 or higher. For older versions of PHP, check out the 0.x-branch.

The recommended way to install PHP-FFMpeg is through Composer.

$ composer require php-ffmpeg/php-ffmpeg

Basic Usage

require 'vendor/autoload.php';

$ffmpeg = FFMpeg\FFMpeg::create();
$video = $ffmpeg->open('video.mpg');
    ->resize(new FFMpeg\Coordinate\Dimension(320, 240))
    ->save(new FFMpeg\Format\Video\X264(), 'export-x264.mp4')
    ->save(new FFMpeg\Format\Video\WMV(), 'export-wmv.wmv')
    ->save(new FFMpeg\Format\Video\WebM(), 'export-webm.webm');


This documentation is an introduction to discover the API. It's recommended to browse the source code as it is self-documented.


FFMpeg\FFMpeg is the main object to use to manipulate medias. To build it, use the static FFMpeg\FFMpeg::create:

$ffmpeg = FFMpeg\FFMpeg::create();

FFMpeg will autodetect ffmpeg and ffprobe binaries. If you want to give binary paths explicitly, you can pass an array as configuration. A Psr\Logger\LoggerInterface can also be passed to log binary executions.

$ffmpeg = FFMpeg\FFMpeg::create(array(
    'ffmpeg.binaries'  => '/opt/local/ffmpeg/bin/ffmpeg',
    'ffprobe.binaries' => '/opt/local/ffmpeg/bin/ffprobe',
    'timeout'          => 3600, // The timeout for the underlying process
    'ffmpeg.threads'   => 12,   // The number of threads that FFMpeg should use
), $logger);

You may pass a temporary_directory key to specify a path for temporary files.

$ffmpeg = FFMpeg\FFMpeg::create(array(
    'temporary_directory' => '/var/ffmpeg-tmp'
), $logger);

Manipulate media

FFMpeg\FFMpeg creates media based on URIs. URIs could be either a pointer to a local filesystem resource, an HTTP resource or any resource supported by FFmpeg.

Note: To list all supported resource type of your FFmpeg build, use the -protocols command:

ffmpeg -protocols

To open a resource, use the FFMpeg\FFMpeg::open method.


Two types of media can be resolved: FFMpeg\Media\Audio and FFMpeg\Media\Video. A third type, FFMpeg\Media\Frame, is available through videos.

View on Github

5 - UnifiedArchive:

A unified reader and writer of compressed archives.

How it works

UnifiedArchive uses "drivers", which can be one of types:

  1. PHP Extensions
  2. Utilities + bridge
  3. Pure PHP

By default, UA goes top-down to select first available driver for passed archive. So, PHP Extension driver will be used (if available), then Utilities + bridge driver (if available), and then Pure PHP driver.

There is at least one driver in all three types, which handles zip format, so this format can be fully supported in any OS/PHP configuration:

  1. zip / phar PHP extensions
  2. Utility + bridge SevenZip / AlchemyZippy
  3. Pure NelexaZip

tar format (with compressed variants) supported by:

  1. phar PHP extension
  2. Utility + bridge SevenZip / AlchemyZippy
  3. Pure TarByPear

So, there is always one driver that supports popular formats, and you should not remember how to work with this concrete driver (zip/phar/SevenZip/AlchemyZippy/NelexaZip), interface for them is uniform.

Quick start

Installation and configuration

Install library

composer require wapmorgan/unified-archive
#Check supported formats with installed drivers
./vendor/bin/cam system:formats
#Check supported functions for zip format
./vendor/bin/cam system:format zip

Install new driver

#Read installation instructions from
./vendor/bin/cam system:drivers
#install missing drivers, for example pear/archive_tar
composer require pear/archive_tar
#check out driver functions
./vendor/bin/cam system:formats TarByPear

#if needed, install extensions, cli tools and php libraries
#to enable support of other formats


use \wapmorgan\UnifiedArchive\UnifiedArchive;

$output_dir = '/var/www/extracted';

# Extraction
$archive = UnifiedArchive::open('archive.zip'); // archive.rar, archive.tar.bz2

if (disk_free_space($output_dir) < $archive->getOriginalSize()) {
    throw new \RuntimeException('No needed space available. Need ' . ($archive->getOriginalSize() - disk_free_space($output_dir)) . ' byte(s) more');

$extracted = $archive->extract($output_dir);
echo 'Number of extracted files' . $extracted.PHP_EOL;

# Archiving
    'README.md' => '/default/path/to/README.md',
    '' => '/folder/with/content/',
], 'archive.zip');

View on Github

Thank you for following this article.

Related videos:

How to Open ZIP Files With PHP

#php #file #type #detection 

5 Favorite PHP Libraries for File Manipulation and MIME Type Detection
Nat  Grady

Nat Grady


AnomalyDetection: Anomaly Detection with R

AnomalyDetection R package

AnomalyDetection is an open-source R package to detect anomalies which is robust, from a statistical standpoint, in the presence of seasonality and an underlying trend. The AnomalyDetection package can be used in wide variety of contexts. For example, detecting anomalies in system metrics after a new software release, user engagement post an A/B test, or for problems in econometrics, financial engineering, political and social sciences.

How the package works

The underlying algorithm – referred to as Seasonal Hybrid ESD (S-H-ESD) builds upon the Generalized ESD test for detecting anomalies. Note that S-H-ESD can be used to detect both global as well as local anomalies. This is achieved by employing time series decomposition and using robust statistical metrics, viz., median together with ESD. In addition, for long time series (say, 6 months of minutely data), the algorithm employs piecewise approximation - this is rooted to the fact that trend extraction in the presence of anomalies in non-trivial - for anomaly detection.

Besides time series, the package can also be used to detect anomalies in a vector of numerical values. We have found this very useful as many times the corresponding timestamps are not available. The package provides rich visualization support. The user can specify the direction of anomalies, the window of interest (such as last day, last hour), enable/disable piecewise approximation; additionally, the x- and y-axis are annotated in a way to assist visual data analysis.

How to get started

Install the R package using the following commands on the R console:


The function AnomalyDetectionTs is called to detect one or more statistically significant anomalies in the input time series. The documentation of the function AnomalyDetectionTs, which can be seen by using the following command, details the input arguments and the output of the function AnomalyDetectionTs.


The function AnomalyDetectionVec is called to detect one or more statistically significant anomalies in a vector of observations. The documentation of the function AnomalyDetectionVec, which can be seen by using the following command, details the input arguments and the output of the function AnomalyDetectionVec.


A simple example

To get started, the user is recommended to use the example dataset which comes with the packages. Execute the following commands:

res = AnomalyDetectionTs(raw_data, max_anoms=0.02, direction='both', plot=TRUE)

Fig 1

From the plot, we observe that the input time series experiences both positive and negative anomalies. Furthermore, many of the anomalies in the time series are local anomalies within the bounds of the time series’ seasonality (hence, cannot be detected using the traditional approaches). The anomalies detected using the proposed technique are annotated on the plot. In case the timestamps for the plot above were not available, anomaly detection could then carried out using the AnomalyDetectionVec function; specifically, one can use the following command:

AnomalyDetectionVec(raw_data[,2], max_anoms=0.02, period=1440, direction='both', only_last=FALSE, plot=TRUE)

Often, anomaly detection is carried out on a periodic basis. For instance, at times, one may be interested in determining whether there was any anomaly yesterday. To this end, we support a flag only_last whereby one can subset the anomalies that occurred during the last day or last hour. Execute the following command:

res = AnomalyDetectionTs(raw_data, max_anoms=0.02, direction='both', only_last=”day”, plot=TRUE)

Fig 2

From the plot, we observe that only the anomalies that occurred during the last day have been annotated. Further, the prior six days are included to expose the seasonal nature of the time series but are put in the background as the window of prime interest is the last day.

Anomaly detection for long duration time series can be carried out by setting the longterm argument to T.

Download Details:

Author: Twitter
Source Code: https://github.com/twitter/AnomalyDetection 
License: GPL-3.0 license

#r #detection 

AnomalyDetection: Anomaly Detection with R
Mike  Kozey

Mike Kozey


Flutter_jailbreak_detection: Flutter Jailbreak Detection Plugin


Flutter jailbreak and root detection plugin.

It uses RootBeer on Android, and DTTJailbreakDetection on iOS.

Getting Started

import 'package:flutter_jailbreak_detection/flutter_jailbreak_detection.dart';

bool jailbroken = await FlutterJailbreakDetection.jailbroken;
bool developerMode = await FlutterJailbreakDetection.developerMode; // android only.


Use this package as a library

Depend on it

Run this command:

With Flutter:

 $ flutter pub add flutter_jailbreak_detection

This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

  flutter_jailbreak_detection: ^1.8.0

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:flutter_jailbreak_detection/flutter_jailbreak_detection.dart';


import 'package:flutter/material.dart';
import 'dart:async';

import 'package:flutter/services.dart';
import 'package:flutter_jailbreak_detection/flutter_jailbreak_detection.dart';

void main() => runApp(new MyApp());

class MyApp extends StatefulWidget {
  _MyAppState createState() => new _MyAppState();

class _MyAppState extends State<MyApp> {
  bool? _jailbroken;
  bool? _developerMode;

  void initState() {

  // Platform messages are asynchronous, so we initialize in an async method.
  Future<void> initPlatformState() async {
    bool jailbroken;
    bool developerMode;
    // Platform messages may fail, so we use a try/catch PlatformException.
    try {
      jailbroken = await FlutterJailbreakDetection.jailbroken;
      developerMode = await FlutterJailbreakDetection.developerMode;
    } on PlatformException {
      jailbroken = true;
      developerMode = true;

    // If the widget was removed from the tree while the asynchronous platform
    // message was in flight, we want to discard the reply rather than calling
    // setState to update our non-existent appearance.
    if (!mounted) return;

    setState(() {
      _jailbroken = jailbroken;
      _developerMode = developerMode;

  Widget build(BuildContext context) {
    return new MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Jailbroken plugin example app'),
        body:  Center(
          child: Column( mainAxisAlignment: MainAxisAlignment.center,
            children: <Widget>[Text('Jailbroken: ${_jailbroken == null ? "Unknown" : _jailbroken! ? "YES" : "NO"}'),
            Text('Developer mode: ${_developerMode == null ? "Unknown" : _developerMode! ? "YES" : "NO"}')

Author: jeroentrappers
Source Code: https://github.com/jeroentrappers/flutter_jailbreak_detection 
License: BSD-3-Clause license

#flutter #dart #detection 

Flutter_jailbreak_detection: Flutter Jailbreak Detection Plugin
Royce  Reinger

Royce Reinger


Cld: Compact Language Detection in Ruby

Compact Language Detection

Blazing-fast language detection for Ruby provided by Google Chrome's Compact Language Detector.

How to Use

CLD.detect_language("This is a test")
# => {:name => "ENGLISH", :code => "en", :reliable => true}

CLD.detect_language("plus ça change, plus c'est la même chose")
# => {:name => "FRENCH", :code => "fr", :reliable => true}


Add this line to your application's Gemfile:

gem "cld"

And then execute:

$ bundle


Thanks to the Chrome authors, and to Mike McCandless for writing a Python version.

Author: jtoy
Source Code: https://github.com/jtoy/cld 
License: BSD-3-Clause license

#ruby #language #detection 

Cld: Compact Language Detection in Ruby

A Fast Golang Library for Media Type & File Extension Detection


A package for detecting MIME types and extensions based on magic numbers

Goroutine safe, extensible, no C bindings    



go get github.com/gabriel-vasile/mimetype


mtype := mimetype.Detect([]byte)
// OR
mtype, err := mimetype.DetectReader(io.Reader)
// OR
mtype, err := mimetype.DetectFile("/path/to/file")
fmt.Println(mtype.String(), mtype.Extension())

See the runnable Go Playground examples.


Only use libraries like mimetype as a last resort. Content type detection using magic numbers is slow, inaccurate, and non-standard. Most of the times protocols have methods for specifying such metadata; e.g., Content-Type header in HTTP and SMTP.


Q: My file is in the list of supported MIME types but it is not correctly detected. What should I do?

A: Some file formats (often Microsoft Office documents) keep their signatures towards the end of the file. Try increasing the number of bytes used for detection with:

mimetype.SetLimit(1024*1024) // Set limit to 1MB.
// or
mimetype.SetLimit(0) // No limit, whole file content used.

If increasing the limit does not help, please open an issue.


mimetype uses a hierarchical structure to keep the MIME type detection logic. This reduces the number of calls needed for detecting the file type. The reason behind this choice is that there are file formats used as containers for other file formats. For example, Microsoft Office files are just zip archives, containing specific metadata files. Once a file has been identified as a zip, there is no need to check if it is a text file, but it is worth checking if it is an Microsoft Office file.

To prevent loading entire files into memory, when detecting from a reader or from a file mimetype limits itself to reading only the header of the input.



Thanks to the hierarchical structure, searching for common formats first, and limiting itself to file headers, mimetype matches the performance of stdlib http.DetectContentType while outperforming the alternative package.

                            mimetype  http.DetectContentType      filetype
BenchmarkMatchTar-24       250 ns/op         400 ns/op           3778 ns/op
BenchmarkMatchZip-24       524 ns/op         351 ns/op           4884 ns/op
BenchmarkMatchJpeg-24      103 ns/op         228 ns/op            839 ns/op
BenchmarkMatchGif-24       139 ns/op         202 ns/op            751 ns/op
BenchmarkMatchPng-24       165 ns/op         221 ns/op           1176 ns/op



Author: Gabriel-vasile
Source Code: https://github.com/gabriel-vasile/mimetype 
License: MIT license

#go #golang #detection 

A Fast Golang Library for Media Type & File Extension Detection

Finger Detection and Tracking using OpenCV and Python

Finger Detection and Tracking

Tracking the movement of a finger is an important feature of many computer vision applications. In this application, A histogram based approach is used to separate out the hand from the background frame. Thresholding and Filtering techniques are used for background cancellation to obtain optimum results.


Finger Detection and Tracking using OpenCV and Python

How to run the code

Go to the Finger Detection and Tracking directory. Then,

  1. Run the code with command python FingerDetection.py
  2. Put your parm over the green squares
  3. Then, press z key to start tracking
  4. Finish program with Esc key.


Demo GitHub Sectory

Stargazers over time

Stargazers over time


You can report the bugs at the issue tracker


You can tweet me if you can't get it to work. In fact, you should tweet me anyway.

Download Details:
Author: amarlearning
Source Code: https://github.com/amarlearning/Finger-Detection-and-Tracking
License: MIT License

#opencv #track #detection 

Finger Detection and Tracking using OpenCV and Python

Fingers Detection using OpenCV and Python


  • OS: MacOS El Capitan
  • Platform: Python 3
  • Librarys:
    • OpenCV 3
    • appscript

How to run it?

  • run it in python
  • press 'b' to capture the background model (Remember to move your hand out of the blue rectangle)
  • press 'r' to reset the backgroud model
  • press 'ESC' to exit


Capture original image

Capture video from camera and pick up a frame.

Alt text

Capture background model & Background subtraction

Use background subtraction method called Gaussian Mixture-based Background/Foreground Segmentation Algorithm to subtract background.

For more information about the method, check Zivkovic2004

Here I use the OpenCV's built-in function BackgroundSubtractorMOG2 to subtract background.

bgModel = cv2.BackgroundSubtractorMOG2(0, bgSubThreshold)

Build a background subtractor model

fgmask = bgModel.apply(frame)

Apply the model to a frame

res = cv2.bitwise_and(frame, frame, mask=fgmask)

Get the foreground(hand) image

Alt text

Gaussian blur & Threshold

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

First convert the image to gray scale.

blur = cv2.GaussianBlur(gray, (blurValue, blurValue), 0)

By Gaussian blurring, we create smooth transition from one color to another and reduce the edge content.

Alt text

ret, thresh = cv2.threshold(blur, threshold, 255, cv2.THRESH_BINARY)

We use thresholding to create binary images from grayscale images.

Alt text

Contour & Hull & Convexity

We now need to find out the hand contour from the binary image we created before and detect fingers (or in other words, recognize gestures)

contours, hierarchy = cv2.findContours(thresh1, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

This function will find all the contours from the binary image. We need to get the biggest contours (our hand) based on their area since we can assume that our hand will be the biggest contour in this situation. (it's obvious)

After picking up our hand, we can create its hull and detect the defects by calling :

hull = cv2.convexHull(res)
defects = cv2.convexityDefects(res, hull)

Alt text

Now we have the number of fingers. How to use this information? It's based on your imagination...

I add in a keyboard simulation package named appscript as interface to control Chrome's dinosaur game.

Alt text

References & Tutorials

  1. OpenCV documentation: http://docs.opencv.org/2.4.13/
  2. Opencv python hand gesture recognition: http://creat-tabu.blogspot.com/2013/08/opencv-python-hand-gesture-recognition.html
  3. Mahaveerverma's hand gesture recognition project: hand-gesture-recognition-opencv

Demo Videos

Download Details:
Author: lzane
Source Code: https://github.com/lzane/Fingers-Detection-using-OpenCV-and-Python
License: MIT License

#opencv #python #detection 

Fingers Detection using OpenCV and Python
Monty  Boehm

Monty Boehm


Sqlmap: Automatic SQL injection and Database Takeover tool


sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester, and a broad range of switches including database fingerprinting, over data fetching from the database, accessing the underlying file system, and executing commands on the operating system via out-of-band connections.



You can visit the collection of screenshots demonstrating some of the features on the wiki.


You can download the latest tarball by clicking here or latest zipball by clicking here.

Preferably, you can download sqlmap by cloning the Git repository:

git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

sqlmap works out of the box with Python version 2.6, 2.7 and 3.x on any platform.


To get a list of basic options and switches use:

python sqlmap.py -h

To get a list of all options and switches use:

python sqlmap.py -hh

You can find a sample run here. To get an overview of sqlmap capabilities, a list of supported features, and a description of all options and switches, along with examples, you are advised to consult the user's manual.



Author: Sqlmapproject
Source Code: https://github.com/sqlmapproject/sqlmap 
License: View license

#python #database #detection 

Sqlmap: Automatic SQL injection and Database Takeover tool
Oral  Brekke

Oral Brekke


is-online: Check If The internet Connection Is Up


Check if the internet connection is up

Works in Node.js and the browser (with a bundler).

In the browser you have navigator.onLine, but it's useless as it only tells you if there's a local connection, and not whether the internet is accessible.


$ npm install is-online


const isOnline = require('is-online');

(async () => {
	console.log(await isOnline());
	//=> true




Type: object


Type: number
Default: 5000

Milliseconds to wait for a server to respond.


Type: number
Values: 4 | 6
Default: 4

Internet Protocol version to use. This is an advanced option that is usually not necessary to be set, but it can prove useful to specifically assert IPv6 connectivity.

How it works

The following checks are run in parallel:

  • Retrieve icanhazip.com (or ipify.org as fallback) via HTTPS.
  • Query myip.opendns.com and o-o.myaddr.l.google.com DNS entries. (Node.js only)
  • Retrieve Apple's Captive Portal test page (this is what iOS does). (Node.js only)

When any check succeeds, the returned Promise is resolved to true.

Proxy support

To make it work through proxies, you need to set up global-agent.



Author: Sindresorhus
Source Code: https://github.com/sindresorhus/is-online 
License: MIT License

#node #browser #detection 

is-online: Check If The internet Connection Is Up
Dexter  Goodwin

Dexter Goodwin


Anomaly Detection Toolkit (ADTK)

Anomaly Detection Toolkit (ADTK)

Anomaly Detection Toolkit (ADTK) is a Python package for unsupervised / rule-based time series anomaly detection.

As the nature of anomaly varies over different cases, a model may not work universally for all anomaly detection problems. Choosing and combining detection algorithms (detectors), feature engineering methods (transformers), and ensemble methods (aggregators) properly is the key to build an effective anomaly detection model.

This package offers a set of common detectors, transformers and aggregators with unified APIs, as well as pipe classes that connect them together into models. It also provides some functions to process and visualize time series and anomaly events.

See https://adtk.readthedocs.io for complete documentation.


Prerequisites: Python 3.5 or later.

It is recommended to install the most recent stable release of ADTK from PyPI.

pip install adtk

Alternatively, you could install from source code. This will give you the latest, but unstable, version of ADTK.

git clone https://github.com/arundo/adtk.git
cd adtk/
git checkout develop
pip install ./


Please see Quick Start for a simple example.

For more detailed examples of each module of ADTK, please refer to Examples section in the documentation or an interactive demo notebook.


Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update unit tests as appropriate.

Please see Contributing for more details.

Author: Arundo
Source Code: https://github.com/arundo/adtk 
License: MPL-2.0 License

#time #detection #python 

Anomaly Detection Toolkit (ADTK)
Alec  Nikolaus

Alec Nikolaus


Finding Outliers

In statistics, an outlier is a data point that differs significantly from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error; the latter are sometimes excluded from the data set. An outlier can cause serious problems in statistical analyses.

Image for post

In this new article we will explain what outliers are and why they are so important, we will see a practical step by step example in Python, 1, 2 and 3 dimensional displays and the use of a general purpose library.

##What are the Outliers?

It is interesting to see the translations of “outlier” — according to its context — in English:

  • Atypical
  • Featured
  • Exceptional
  • Abnormal
  • Extreme Value, Abnormal Value, Aberrant Value!

That gives us an idea, doesn’t it?

That is, the outliers in our dataset will be the values that “escape the range where most samples are concentrated”. According to Wikipedia they are the samples that are distant from other observations.

Box plot of data from the Michelson–Morley experiment displaying four outliers in the middle column, as well as one outlier in the first column.

Detection of Outliers

And why are we interested in detecting these Outliers? Because they can considerably affect the results that a Machine Learning model can obtain… For bad… or for good! That’s why we have to detect them, and take them into account. For example in Linear Regression or Assembly algorithms can have a negative impact on your predictions.

Good Outliers vs. Bad Outliers

Outliers can mean a number of things:

  • ERROR: If we have a “people age” group and we have a 160 year old person, it’s probably a data loading error. In this case, detecting outliers helps us to detect errors.
  • LIMITS: In other cases, we can have values that escape from the “middle group”, but we want to keep the data modified, so that it does not harm the learning of the ML model.
  • Point of Interest: maybe the “anomalous” cases are the ones we want to detect and they are our target (and not our enemy!)

Many times it is easy to identify the outliers in graphs. Let’s see examples of outliers in 1, 2 and 3 dimensions.

#detection #outliers #anomaly #python

Finding Outliers
Agnes  Sauer

Agnes Sauer


Attack Pattern Detection and Prediction

Cyber-adversaries are becoming more sophisticated in their efforts to avoid detection, and many modern malware tools are already incorporating new ways to bypass antivirus and other threat detection measures. Because networks and organizations use sophisticated methods to detect and respond to attacks, the response can be so strong that criminals try to respond with something even stronger. The complexity of cybercriminals is increasing, combined with the widening potential of artificial intelligence (AI) attacks.

Cybersecurity, however, is at a critical juncture, and the field must focus future research efforts on cyber-attack prediction systems that can anticipate critical scenarios and outcomes, rather than relying on defensive solutions and focusing on mitigation. Computer systems around the world need systems based on a comprehensive, predictive analysis of cyber threats.

Image for post

Artificial intelligence (AI), which relies heavily on machine learning (ML), has the ability to recognize patterns arising from past experiences and make predictions based on them. In recent years, swarm technology, which can use things like machine learning and artificial intelligence to attack networks and devices, has shown new potential.

Useful patterns of attack can be defined by understanding patterns of behavior, analyzing patterns and connections between malicious activities, predicting future moves, and ultimately preventing or detecting potentially malicious behavior.

The aforementioned cyber-threat prediction systems offer promising and limited possibilities, but large-scale coordinated attacks require progress on several fronts, including the detection and prediction of events generated in computer systems. Obfuscation techniques are used to bypass detection by deliberately making malicious code difficult to understand in order to bypass the detection of the network.

When assessing network security risks, hackers’ behavior must be taken into account, which can be a daunting task, given the number of known vulnerabilities and the choices an attacker could make to infiltrate a network.

#machine-learning #detection #prevention #deep learning

Attack Pattern Detection and Prediction
Marc  Schroeder

Marc Schroeder


Edge Detection OpenCV | Laplacian Sobel and Canny Edge Detection using OpenCV Python

This video explains the concepts of Canny, Laplacian and Sobel Edge Detection in Python OpenCV. In this video we will see how to implement all the three edge detection technique using Python. Edge detection is an image processing technique for finding the boundaries of objects within images. It works by detecting discontinuities in brightness and It helps
us reduce the amount of data (pixels) to process and maintains the structural aspect of the image.Edge detection is used for image segmentation and data extraction in areas such as image processing, computer vision, and machine learning. This is the next video in the Python OpenCV Crash Course. Later on, in the upcoming videos, we will see how can we build face detection, object detection types of Computer Vision Projects.

#opencv #python #detection

Edge Detection OpenCV | Laplacian Sobel and Canny Edge Detection using OpenCV Python
Rusty  Bernier

Rusty Bernier


OpenCV Selective Search for Object Detection - PyImageSearch

In this tutorial you will learn how to use OpenCV Selective Search for object detection with Python.

#opencv #selective #object #detection #pyimagesearch

OpenCV Selective Search for Object Detection - PyImageSearch