Willow: A Practical, Open Source, Privacy-Focused Platform

Hello Willow Users!

Many users across various forums, social media, etc are starting to receive their hardware! I have enabled Github discussions to centralize these great conversations - stop by, introduce yourself, and let us know how things are going with Willow! Between Github discussions and issues we can all work together to make sure our early adopters have the best experience possible!

Willow - A Practical, Open Source, Privacy-Focused Platform for Voice Assistants and other Applications

Willow is an ESP IDF based project primarily targetting the ESP BOX hardware from Espressif. Our goal is to provide Amazon Echo/Google Home competitive performance, accuracy, cost, and functionality with Home Assistant and other platforms - 100% open source and completely self-hosted by the user with "ready for the kitchen counter" low cost commercially available hardware.

FAST - Watch the demo. Response times faster than Alexa/Echo or Google Home. From end of speech to action completed in 500ms or less.

ACCURATE - High wake word accuracy, low false activation, and speech recognition powered by our Willow Inference Server (coming soon) or command recognition solely on the device.

RELIABLE - We've tested thousands of cycles of voice commands with a < 1% failure rate. No one likes to repeat themselves!

FLEXIBLE - Use Willow Inference Server anywhere or don't use it at all with command recognition on the device. Have the results go anywhere you want. Integrate with whatever you want. Completely open source so it does what you want, only what you want, and only how you want it. No more annoying extra prompts or sales pitches to upsell you. Supports multiple wake words with more coming soon.

PRIVATE - Check the source. Build and flash yourself. Proxy through another server to inspect traffic. Use on your own server. Use only local commands. Use on a network without access to the internet. Dig as deep as you want because you're not going to find anything fishy here!

PRACTICAL AND NOT UGLY - Ready to go! Take it out of the box, flash, and put it in your home or office in minutes without getting looks from people wondering what that "thing" is. Install as many as you like.

CHEAP - Approximately $50 hardware cost (plus USB-C power supply). Fully assembled. Done.

LOW POWER - 100mW power usage.

Current supported features include:

• Wake Word Engine. Say "Hi ESP" or "Alexa" (user selectable) and start talking!
• Voice Activity Detection. When you stop talking it will stop recording and take action.
• Support for Home Assistant! Simply configure Willow with your Home Assistant server address and access token.
• Support for other platforms. As long as your configured endpoint can take an HTTP POST you can do anything with the speech output!
• Good far-field performance. We've tested wake and speech recognition from roughly 25 feet away in challenging environments with good results.
• Good audio quality - Willow provides features such as automatic gain control, noise separation, etc.
• Support for challenging Wi-Fi environments. Willow can (optionally) use audio compression to reduce airtime on 2.4 GHz Wi-Fi in cases where it's very busy.
• LCD and touchscreen. The ESP BOX has color LCD and capacitive mult-point touchscreen. We support them with an initial user interface.
• Completely on device speech command recognition and support for our (soon to be released) open source Willow Inference Server (Tovera hosted best-effort WIS provided). Configure up to 400 commands completely on device or self-host our (coming soon) inference server to transcribe any speech!

All with hardware you can order today from Amazon, Adafruit, The Pi Hut, Mouser, or other preferred vendor for (approximately) $50 USD. Add a USB-C power supply and go!

Getting Started

Configuring and building Willow for the ESP BOX is a multi-step process. We're working on improving that but for now...

System Dependencies

We use tio as a serial monitor so you will need to install that.

Ubuntu/Debian:

sudo apt-get install tio

Arch Linux:

sudo yay -S tio

Mac (with homebrew):

brew install tio

Clone this repo

git clone https://github.com/toverainc/willow.git && cd willow

Container

We use Docker (also supports podman) for the build container. To build the container with docker:

./utils.sh build-docker

Once the container has finished building you will need to enter it for all following commands:

./utils.sh docker

Install

Once inside the container install the environment:

./utils.sh install

Start Configuration

Start the config process:

./utils.sh config

ESP BOX LITE NOTE: FOR USERS WHO PURCHASED THE ESP BOX LITE!!!

You will need to build for the ESP BOX LITE. From the main menu, select: Audio HAL ---> Audio Board ---> ESP32-S3-BOX-Lite

Return to main menu and continue.

Willow Configuration

Navigate to "Willow Configuration" to fill in your Wi-Fi SSID, Wi-Fi password (supports WPA/WPA2/WPA3), and your Willow server URI (best-effort Tovera hosted example provided).

For Home Assistant you will also need to create a long lived access token and configure your server address. By default we use homeassistant.local which should use mDNS to resolve your local Home Assistant instance. Put your long lived access token in the text input area. We recommend testing both your Home Assistant server address and token before flashing.

If your Home Assistant server requires TLS make sure to select it.

There are also various other configuration options for speaker volume, display brightness, NTP, etc.

If you want to change the wake word from the default "Hi ESP" you can navigate from the main menu to ESP Speech Recognition --> Select wake words ---> and select Alexa or whichever. NOTE: If changing the wake word ALWAYS use the wn9 variants.

Once you've provided those press 'q'. When prompted to save, do that.

Build and exit container

./utils.sh build

When the build completes successfully you can exit the container.

Connect the ESP BOX

It's getting real now - plug it in!

Back on the host - set serial port

To do anything involving the serial port you will need to set the PORT environment variable for all further invocations of utils.sh.

With recent versions of tio you can use tio -L to list available ports. On Linux you can check dmesg and look for the path of the recently connected ESP BOX (furthest at the bottom, hopefully). On Linux it's /dev/ACM* and on Mac it's /dev/usbmodem*.

Examples:

Linux:

export PORT=/dev/ttyACM0

Mac:

export PORT=/dev/cu.usbmodem2101

Flash

For out of the box/factory new ESP BOX hardware you will need to (one time) erase the factory flash before flashing Willow:

./utils.sh erase-flash

Once you have done that you can flash:

./utils.sh flash

It should flash and connect you to the serial monitor.

Let's talk!

If you have made it this far - congratulations! You will see serial monitor output ending like this:

I (10414) AFE_SR: afe interface for speech recognition

I (10424) AFE_SR: AFE version: SR_V220727

I (10424) AFE_SR: Initial auido front-end, total channel: 3, mic num: 2, ref num: 1

I (10434) AFE_SR: aec_init: 1, se_init: 1, vad_init: 1

I (10434) AFE_SR: wakenet_init: 1

MC Quantized wakenet9: wakeNet9_v1h24_hiesp_3_0.63_0.635, tigger:v3, mode:2, p:0, (May  5 2023 20:32:52)
I (10704) AFE_SR: wake num: 3, mode: 1, (May  5 2023 20:32:52)

I (13:26:42.433) AUDIO_THREAD: The feed_task task allocate stack on external memory
I (13:26:42.434) AUDIO_THREAD: The fetch_task task allocate stack on external memory
I (13:26:42.442) AUDIO_THREAD: The recorder_task task allocate stack on external memory
I (13:26:42.451) WILLOW: app_main() - start_rec() finished
I (13:26:42.457) AUDIO_THREAD: The at_read task allocate stack on external memory
I (13:26:42.466) WILLOW: esp_netif_get_nr_of_ifs: 1
I (13:26:42.471) WILLOW: Startup complete. Waiting for wake word.

You should see some help text on the display to use your configured wake word. Try some built in Home Assistant intents like:

• "(Your wake word) Turn on bedroom lights"
• "(Your wake word) Turn off kitchen lights"

The available commands and specific names, etc will depend on your Home Assistant configuration.

You can also provide free-form speech to get an idea of the accuracy and speed provided by our inference server implementation. The commands will fail unless you've defined them in Home Assistant but the display will show the speech recognition results to get your imagination going.

You can now repeat the erase and flash process for as many devices as you want!

Exit serial monitor

To exit tio you need to press 'CTRL+t' and then 'q'. Or you can unplug your device and tio will wait until you reconnect it.

Start serial monitor

If you want to see what your device is up to you can start the serial monitor anytime:

./utils.sh monitor

Things went sideways - reset

In the event your environment gets out of whack we have a helper to reset:

./utils.sh destroy

As the plentiful messages indicate it's a destructive process but it will reset your environment. After it completes you can start from the top and try again.

Recover from a bad flash

ESP devices are very robust to flashing failures but it can happen! If you end up "bricking" your device you can erase the flash:

./utils.sh erase-flash

NOTE: Depending on how tight of a boot loop your device is in you may need to run erase-flash multiple times to get the timing right. It will eventually "catch" and successfully erase the flash. When it reports successful erase you can flash again:

./utils.sh flash

Advanced Usage

utils.sh will attempt to load environment variables from .env. You can define your PORT here to avoid needing to define it over and over.

The ESP-IDF, ESP-ADF, ESP-SR, LVGL, etc libraries have a plethora of configuration options. DO NOT change anything outside of "Willow Configuration" (other than wake word) unless you know what you are doing.

If you want to quickly and easily flash multiple devices or distribute a combined firmware image you can use the dist arguments to utils.sh:

./utils.sh dist - builds the combined flash image (willow-dist.bin)

./utils.sh flash-dist - flashes the combined flash image

This combined firmware image can be used with any ESP flashing tool like the web flasher ESP Tool so you can send firmware images to your less technical friends! Just make sure to erase the flash first and use offset 0x0 with those tools as we include the bootloader.

Development

Development usually involves a few steps:

1. Code - do your thing!
2. Build
3. Flash

Unless you change the wake word and/or are using local command recognition (Multinet) you can selectively flash only the application partition. This avoids long flash times with the wakenet and multinet model partition, etc:

./utils.sh build

./utils.sh flash-app

The Future (in no particular order)

Multiple Languages

Willow supports UTF characters and our Willow Inference Server supports all the languages of Whisper. We have some polishing to do here but it is coming very soon. For the interface language on device we're looking for translation help!

Performance Improvements

Willow and Willow Inference Server/Multinet already provide "faster-than-Alexa" responsiveness for a voice user interface. However, there are multiple obvious optimizations we're aware of:

• ESP ADF pipeline handing (we're waiting on ESP-ADF 2.6 with ESP-IDF 5)
• Websockets for inference server (avoids TLS handshake and connection establishment for each session)
• Code in general (we're new to ESP IDF and it could use review)
• Various performance-related sdkconfig parameters (again, we're new to ESP IDF)
• Likely many, many more

These enhancements alone should dramatically improve responsiveness.

No CUDA

The Whisper Inference Server (open source release soon) will run CPU only but the performance on CPU is not comparable to heavily optimized implementations like whisper.cpp. For an Alexa/Echo competitive voice interface we currently believe that our approach with CUDA or local Multinet (for limited commands) is the best approach. However, we also understand that isn't practical or preferred for many users. Between on device Multinet command recognition and further development on CPU-only Whisper implementations, ROCm, etc we will get there. That said, if you can make the audio streaming API work you can use any speech to text and text to speech implementation you want!

TTS Output

Given the capabilities of Whisper speech commands like "What is the weather in Sofia, Bulgaria?" are transcribed but need to match a command (like a Home Assistant intent) on the destination. Our inference server implementation has a text to speech engine and Home Assistant has a variety of options as well. In the event the final response to a given command results in audio output we can play that via the speakers in the ESP BOX (not yet supported).

Higher Quality Audio Output

The ESP BOX supports bluetooth. In applications where higher quality audio is desired (music streaming, etc) we can support pairing to bluetooth speaker devices. Who knows? Eventually we may even design our own device with better internal speakers...

LCD and Touchscreen Improvements

The ESP BOX has a multi-point capacitive touchscreen and support for many GUI elements. We currently only provide basic features like touch screen to wake up, a little finger cursor thing, and a Cancel button to cancel/interrupt command streaming. There's a lot more work to do here!

Buttons

The ESP BOX has buttons and who doesn't like configuring buttons to do things?!

Audio on device

We currently beep once for success and twice for failure. It's not the most annoying beep in the world but it's not exactly pleasant either. We're going to include some pleasant chimes for success and failure as well as some basic status reporting like "Could not connect to server", etc.

Easy Start

Docker, building, configuring, flashing, etc is a pain. There are several approaches we plan to take to avoid this and ease the barrier to entry for users to get started.

Dynamic Configuration

With something like a Willow Home Assistant component and websocket support we can enable all kinds of interesting dynamic configuration updates and tighter overall configurations.

Over the Air Firmware Updates

ESP IDF and ESP BOX has robust support for over the air firmware (OTA) updates. Down the road we will support them.

Multiple Devices

The good news is the far-field wake word recognition and speech recognition performance is very good. The bad news is if you have multiple devices in proximity they are all likely to wake and process speech simultaneously. Commands will still work but multiple confirmation/error beeps and hammering your destination command endpoint is less than ideal. We have a few ideas about dealing with this too.

Custom Wake Word

Espressif has a wake word customization service that enables us (and commercial users) to create custom wake words. We plan to create a "Hi Willow" or similar wake word and potentially others depending on input from the community.

GPIO

The ESP BOX provides 16 GPIOs to the user that are readily accessed from sockets on the rear of the device. We plan to make these configurable by the user to enable all kinds of interesting maker/DIY functions.

Download Details:

Author: Toverainc
Source Code: https://github.com/toverainc/willow 
License: Apache-2.0 license

#c #privacy #deeplearning #echo #esp32 #speech #recognition 

Rudder-server: Privacy & Security Focused Segment-alternative

The Customer Data Platform for Developers

As the leading open source Customer Data Platform (CDP), RudderStack provides data pipelines that make it easy to collect data from every application, website and SaaS platform, then activate it in your warehouse and business tools.

With RudderStack, you can build customer data pipelines that connect your whole customer data stack and then make them smarter by triggering enrichment and activation in customer tools based on analysis in your data warehouse. It's easy-to-use SDKs and event source integrations, Cloud Extract integrations, transformations, and expansive library of destination and warehouse integrations makes building customer data pipelines for both event streaming and cloud-to-warehouse ELT simple.

Key features

Warehouse-first: RudderStack treats your data warehouse as a first class citizen among destinations, with advanced features and configurable, near real-time sync.

Developer-focused: RudderStack is built API-first. It integrates seamlessly with the tools that the developers already use and love.

High Availability: RudderStack comes with at least 99.99% uptime. We have built a sophisticated error handling and retry system that ensures that your data will be delivered even in the event of network partitions or destinations downtime.

Privacy and Security: You can collect and store your customer data without sending everything to a third-party vendor. With RudderStack, you get fine-grained control over what data to forward to which analytical tool.

Unlimited Events: Event volume-based pricing of most of the commercial systems is broken. With RudderStack Open Source, you can collect as much data as possible without worrying about overrunning your event budgets.

Segment API-compatible: RudderStack is fully compatible with the Segment API. So you don't need to change your app if you are using Segment; just integrate the RudderStack SDKs into your app and your events will keep flowing to the destinations (including data warehouses) as before.

Production-ready: Companies like Mattermost, IFTTT, Torpedo, Grofers, 1mg, Nana, OnceHub, and dozens of large companies use RudderStack for collecting their events.

Seamless Integration: RudderStack currently supports integration with over 90 popular tool and warehouse destinations.

User-specified Transformation: RudderStack offers a powerful JavaScript-based event transformation framework which lets you enhance or transform your event data by combining it with your other internal data. Furthermore, as RudderStack runs inside your cloud or on-premise environment, you can easily access your production data to join with the event data.

Get started

The easiest way to experience RudderStack is to sign up for RudderStack Cloud Free - a completely free tier of RudderStack Cloud.

You can also set up RudderStack on your platform of choice with these two easy steps:

Step 1: Set up RudderStack

• Docker
• Kubernetes
• Developer machine setup

Note: If you are planning to use RudderStack in production, we STRONGLY recommend using our Kubernetes Helm charts. We update our Docker images with bug fixes much more frequently than our GitHub repo.

Step 2: Verify the installation

Once you have installed RudderStack, send test events to verify the setup.

Architecture

RudderStack is an independent, stand-alone system with a dependency only on the database (PostgreSQL). Its backend is written in Go with a rich UI written in React.js.

A high-level view of RudderStack’s architecture is shown below:

For more details on the various architectural components, refer to our documentation.

Contribute

We would love to see you contribute to RudderStack. Get more information on how to contribute here.

Download Details:

Author: Rudderlabs
Source Code: https://github.com/rudderlabs/rudder-server/ 
License: AGPL-3.0 license

#golang #react #bigquery #security #privacy 

Offen Fair Web Analytics

 Offen 

Fair web analytics

Let your users access their data. Gain valuable insights at the same time. Open, lightweight, self hosted and free.

Community

• Feedback and contributions welcome
• Kind support
• Who's using Offen?
• Links

Core features

Secure & free
Our code is open source. All usage data is encrypted end-to-end. Offen will always be available for free.

Self hosted
Comply with GDPR guidelines. No ads, no third parties involved. Offen uses first-party cookies only.

Fair & by choice
Opt-in only. Users must actively give their consent to data collection. They have full access to their data.

How it works

Your job

• Self host Offen while protecting your users' data.
• Integrate the code snippet into pages you want to track.
• Make your users aware of the access to their data.
• Improve your services with fair and transparent insights.

Benefits for your users

• Opt in to data collection or decide to not participate at all.
• Review own data with detailed explanations of metrics and terms.
• Only delete usage data or opt out completely at any time.

What you see
Data of all pages where your Offen installation is active. For example:

What your users see
Data of all pages a user has visited where your Offen installation is active. For example:

More features

• Easily analyze multiple websites within one installation.
• All website accounts can be shared within teams.
• User data is only stored for 6 months and then deleted.
• A detailed documentation on how to run Offen is available.

Essential metrics

All important statistics that help you to improve your service.
Filter collected data by URL, Location, Referrer, UTM parameters, as well as Landing Pages and Exit Pages.

Objectives

Privacy friendly
Collection of usage data is opt in, users that do not actively opt in will never leave a trace. After opt in, Offen collects the minimal amount of data needed to generate meaningful statistics for operators. No IPs, User-Agent strings or similar are being collected or even looked at.

Secure
Data in Offen is encrypted End-To-End. Clients encrypt usage data before it leaves the browser and there is no way for the server storing this data to decrypt it. Attackers have no means to compromise an instance, accidental data leaks cannot expose user data.

Self hosted and lightweight
You can run Offen on-premises, or in any other deployment scenario that fits your need. All you need to do is download a single binary file or pull a Docker image, and run it on your server. Offen will automatically install and renew SSL certificates for you if you want it to. If you do not want to deploy a dedicated database, you can use SQLite to store data directly on the server.

Transparent and fair
Offen treats the user as a party of equal importance in the collection of usage data. Users have access to the same set of tools for analyzing their own data and they can delete their data at any time.

Localize

Offen is currently available in English, German, French, Spanish and Portuguese. Our consent banner and the Auditorium for operators as well as users can be displayed in the respective locale.

Support fair web analytics by contributing Italian, Chinese and other language versions. Request an invite.

Customize

The Offen consent banner is customizable in color, shape and basic font specifications to match your design. Learn more.

Test drive

Give Offen a test drive right now. Open your terminal and type:

curl https://demo.offen.dev | bash

This creates an ephemeral one-off installation that is populated with random data and is running on http://localhost:9876. There, you can log in using the account demo@offen.dev and password demo.

License

Work in this repository is licensed under multiple licences.

• All original source code is licensed under Apache-2.0.
• All documentation is licensed under Apache-2.0.
• The Offen icon and logo are licensed as CC-BY-NC-ND-4.0.

See the NOTICE file for license information on works that get bundled by Offen.

Project status

Offen is in active development.
Check our roadmap and blog for detailed updates on what we are working on right now and what's up next.

Feedback and contributions welcome

Found an issue or want to add something? Please do not hesitate to file an issue or open a pull request (or send an email in case you don't want to use GitHub). For details on how to get started head over to our contributing guidelines.

This repository contains all source code needed to work on Offen, both on the server as well as on the client. The development setup requires docker and docker-compose to be installed.

After cloning the repository

$ git clone git@github.com:offen/offen.git

you can build the containers and install dependencies using:

$ make setup

Next seed the database for the server application:

$ make bootstrap

You can test your setup by starting the application:

$ make up

which should enable you to access http://localhost:8080/auditorium/ and use the Auditorium

Run the tests for all subapplications using

$ make test

Detailed instructions on how to get started with development can be found at our dedicated docs site.

The documentation site at https://docs.offen.dev is also part of this repository. To run this site locally, you can:

make setup-docs
make docs

This will serve the documentation site on https://localhost:4000.

Who's using Offen?

Are you using Offen? We're happy to feature you in this README. Send a PR adding your site or app to this section.

Links

Website
Docs
Twitter
Mastodon

Download Details:

Author: offen
Source Code: https://github.com/offen/offen 
License: Apache-2.0 license

#javascript #privacy #webanalytics 

GoAccess: A Real-time Web Log analyzer & interactive Viewer

GoAccess 

What is it?

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through your browser. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. More info at: https://goaccess.io.

Features

GoAccess parses the specified web log file and outputs the data to the X terminal. Features include:

Completely Real Time
All panels and metrics are timed to be updated every 200 ms on the terminal output and every second on the HTML output.

Minimal Configuration needed
You can just run it against your access log file, pick the log format and let GoAccess parse the access log and show you the stats.

Track Application Response Time
Track the time taken to serve the request. Extremely useful if you want to track pages that are slowing down your site.

Nearly All Web Log Formats
GoAccess allows any custom log format string. Predefined options include, Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, etc.

Incremental Log Processing
Need data persistence? GoAccess has the ability to process logs incrementally through the on-disk persistence options.

Only one dependency
GoAccess is written in C. To run it, you only need ncurses as a dependency. That's it. It even features its own Web Socket server — http://gwsocket.io/.

Visitors
Determine the amount of hits, visitors, bandwidth, and metrics for slowest running requests by the hour, or date.

Metrics per Virtual Host
Have multiple Virtual Hosts (Server Blocks)? It features a panel that displays which virtual host is consuming most of the web server resources.

ASN (Autonomous System Number mapping)
Great for detecting malicious traffic patterns and block them accordingly.

Color Scheme Customizable
Tailor GoAccess to suit your own color taste/schemes. Either through the terminal, or by simply applying the stylesheet on the HTML output.

Support for Large Datasets
GoAccess features the ability to parse large logs due to its optimized in-memory hash tables. It has very good memory usage and pretty good performance. This storage has support for on-disk persistence as well.

Docker Support
Ability to build GoAccess' Docker image from upstream. You can still fully configure it, by using Volume mapping and editing goaccess.conf. See Docker section below.

Nearly all web log formats...

GoAccess allows any custom log format string. Predefined options include, but not limited to:

• Amazon CloudFront (Download Distribution).
• Amazon Simple Storage Service (S3)
• AWS Elastic Load Balancing
• Combined Log Format (XLF/ELF) Apache | Nginx
• Common Log Format (CLF) Apache
• Google Cloud Storage.
• Apache virtual hosts
• Squid Native Format.
• W3C format (IIS).
• Caddy's JSON Structured format.

Why GoAccess?

GoAccess was designed to be a fast, terminal-based log analyzer. Its core idea is to quickly analyze and view web server statistics in real time without needing to use your browser (great if you want to do a quick analysis of your access log via SSH, or if you simply love working in the terminal).

While the terminal output is the default output, it has the capability to generate a complete, self-contained, real-time HTML report, as well as a JSON, and CSV report.

You can see it more of a monitor command tool than anything else.

Installation

Build from release

GoAccess can be compiled and used on *nix systems.

Download, extract and compile GoAccess with:

$ wget https://tar.goaccess.io/goaccess-1.7.1.tar.gz
$ tar -xzvf goaccess-1.7.1.tar.gz
$ cd goaccess-1.7.1/
$ ./configure --enable-utf8 --enable-geoip=mmdb
$ make
# make install

Build from GitHub (Development)

$ git clone https://github.com/allinurl/goaccess.git
$ cd goaccess
$ autoreconf -fiv
$ ./configure --enable-utf8 --enable-geoip=mmdb
$ make
# make install

Build in isolated container

You can also build the binary for Debian based systems in an isolated container environment to prevent cluttering your local system with the development libraries:

$ curl -L "https://github.com/allinurl/goaccess/archive/refs/heads/master.tar.gz" | tar -xz && cd goaccess-master
$ docker build -t goaccess/build.debian-10 -f Dockerfile.debian-10 .
$ docker run -i --rm -v $PWD:/goaccess goaccess/build.debian-10 > goaccess

Distributions

It is easiest to install GoAccess on GNU+Linux using the preferred package manager of your GNU+Linux distribution. Please note that not all distributions will have the latest version of GoAccess available.

Debian/Ubuntu

# apt-get install goaccess

Note: It is likely this will install an outdated version of GoAccess. To make sure that you're running the latest stable version of GoAccess see alternative option below.

Official GoAccess Debian & Ubuntu repository

$ wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor \
    | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
$ echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=$(dpkg --print-architecture)] https://deb.goaccess.io/ $(lsb_release -cs) main" \
    | sudo tee /etc/apt/sources.list.d/goaccess.list
$ sudo apt-get update
$ sudo apt-get install goaccess

Note:

• .deb packages in the official repo are available through HTTPS as well. You may need to install apt-transport-https.

Fedora

# yum install goaccess

Arch

# pacman -S goaccess

Gentoo

# emerge net-analyzer/goaccess

OS X / Homebrew

# brew install goaccess

FreeBSD

# cd /usr/ports/sysutils/goaccess/ && make install clean
# pkg install sysutils/goaccess

OpenBSD

# cd /usr/ports/www/goaccess && make install clean
# pkg_add goaccess

openSUSE

# zypper ar -f obs://server:http http
# zypper in goaccess

OpenIndiana

# pkg install goaccess

pkgsrc (NetBSD, Solaris, SmartOS, ...)

# pkgin install goaccess

Windows

GoAccess can be used in Windows through Cygwin. See Cygwin's packages. Or through the GNU+Linux Subsystem on Windows 10.

Distribution Packages

GoAccess has minimal requirements, it's written in C and requires only ncurses. However, below is a table of some optional dependencies in some distros to build GoAccess from source.

Note: You may need to install build tools like gcc, autoconf, gettext, autopoint etc for compiling/building software from source. e.g., base-devel, build-essential, "Development Tools".

Docker

A Docker image has been updated, capable of directing output from an access log. If you only want to output a report, you can pipe a log from the external environment to a Docker-based process:

cat access.log | docker run --rm -i -e LANG=$LANG allinurl/goaccess -a -o html --log-format COMBINED - > report.html

OR real-time

tail -F access.log | docker run -p 7890:7890 --rm -i -e LANG=$LANG allinurl/goaccess -a -o html --log-format COMBINED --real-time-html - > report.html

You can read more about using the docker image in DOCKER.md.

Storage

Default Hash Tables

In-memory storage provides better performance at the cost of limiting the dataset size to the amount of available physical memory. GoAccess uses in-memory hash tables. It has very good memory usage and pretty good performance. This storage has support for on-disk persistence as well.

Command Line / Config Options

See options that can be supplied to the command or specified in the configuration file. If specified in the configuration file, long options need to be used without prepending --.

Usage / Examples

Note: Piping data into GoAccess won't prompt a log/date/time configuration dialog, you will need to previously define it in your configuration file or in the command line.

Getting Started

To output to a terminal and generate an interactive report:

# goaccess access.log

To generate an HTML report:

# goaccess access.log -a > report.html

To generate a JSON report:

# goaccess access.log -a -d -o json > report.json

To generate a CSV file:

# goaccess access.log --no-csv-summary -o csv > report.csv

GoAccess also allows great flexibility for real-time filtering and parsing. For instance, to quickly diagnose issues by monitoring logs since goaccess was started:

# tail -f access.log | goaccess -

And even better, to filter while maintaining opened a pipe to preserve real-time analysis, we can make use of tail -f and a matching pattern tool such as grep, awk, sed, etc:

# tail -f access.log | grep -i --line-buffered 'firefox' | goaccess --log-format=COMBINED -

or to parse from the beginning of the file while maintaining the pipe opened and applying a filter

# tail -f -n +0 access.log | grep -i --line-buffered 'firefox' | goaccess -o report.html --real-time-html -

Multiple Log files

There are several ways to parse multiple logs with GoAccess. The simplest is to pass multiple log files to the command line:

# goaccess access.log access.log.1

It's even possible to parse files from a pipe while reading regular files:

# cat access.log.2 | goaccess access.log access.log.1 -

Note: the single dash is appended to the command line to let GoAccess know that it should read from the pipe.

Now if we want to add more flexibility to GoAccess, we can use zcat --force to read compressed and uncompressed files. For instance, if we would like to process all log files access.log*, we can do:

# zcat --force access.log* | goaccess -

Note: On Mac OS X, use gunzip -c instead of zcat.

Real-time HTML outputs

GoAccess has the ability the output real-time data in the HTML report. You can even email the HTML file since it is composed of a single file with no external file dependencies, how neat is that!

The process of generating a real-time HTML report is very similar to the process of creating a static report. Only --real-time-html is needed to make it real-time.

# goaccess access.log -o /usr/share/nginx/html/your_site/report.html --real-time-html

To view the report you can navigate to http://your_site/report.html.

By default, GoAccess will use the host name of the generated report. Optionally, you can specify the URL to which the client's browser will connect to. See FAQ for a more detailed example.

# goaccess access.log -o report.html --real-time-html --ws-url=goaccess.io

By default, GoAccess listens on port 7890, to use a different port other than 7890, you can specify it as (make sure the port is opened):

# goaccess access.log -o report.html --real-time-html --port=9870

And to bind the WebSocket server to a different address other than 0.0.0.0, you can specify it as:

# goaccess access.log -o report.html --real-time-html --addr=127.0.0.1

Note: To output real time data over a TLS/SSL connection, you need to use --ssl-cert=<cert.crt> and --ssl-key=<priv.key>.

Filtering

Working with dates

Another useful pipe would be filtering dates out of the web log

The following will get all HTTP requests starting on 05/Dec/2010 until the end of the file.

# sed -n '/05\/Dec\/2010/,$ p' access.log | goaccess -a -

or using relative dates such as yesterdays or tomorrows day:

# sed -n '/'$(date '+%d\/%b\/%Y' -d '1 week ago')'/,$ p' access.log | goaccess -a -

If we want to parse only a certain time-frame from DATE a to DATE b, we can do:

# sed -n '/5\/Nov\/2010/,/5\/Dec\/2010/ p' access.log | goaccess -a -

If we want to preserve only certain amount of data and recycle storage, we can keep only a certain number of days. For instance to keep & show the last 5 days:

# goaccess access.log --keep-last=5

Virtual hosts

Assuming your log contains the virtual host field. For instance:

vhost.io:80 8.8.4.4 - - [02/Mar/2016:08:14:04 -0600] "GET /shop HTTP/1.1" 200 615 "-" "Googlebot-Image/1.0"

And you would like to append the virtual host to the request in order to see which virtual host the top urls belong to:

awk '$8=$1$8' access.log | goaccess -a -

To do the same, but also use real-time filtering and parsing:

tail -f  access.log | unbuffer -p awk '$8=$1$8' | goaccess -a -

To exclude a list of virtual hosts you can do the following:

# grep -v "`cat exclude_vhost_list_file`" vhost_access.log | goaccess -

Files, status codes and bots

To parse specific pages, e.g., page views, html, htm, php, etc. within a request:

# awk '$7~/\.html|\.htm|\.php/' access.log | goaccess -

Note, $7 is the request field for the common and combined log format, (without Virtual Host), if your log includes Virtual Host, then you probably want to use $8 instead. It's best to check which field you are shooting for, e.g.:

# tail -10 access.log | awk '{print $8}'

Or to parse a specific status code, e.g., 500 (Internal Server Error):

# awk '$9~/500/' access.log | goaccess -

Or multiple status codes, e.g., all 3xx and 5xx:

# tail -f -n +0 access.log | awk '$9~/3[0-9]{2}|5[0-9]{2}/' | goaccess -o out.html -

And to get an estimated overview of how many bots (crawlers) are hitting your server:

# tail -F -n +0 access.log | grep -i --line-buffered 'bot' | goaccess -

Tips

Also, it is worth pointing out that if we want to run GoAccess at lower priority, we can run it as:

# nice -n 19 goaccess -f access.log -a

and if you don't want to install it on your server, you can still run it from your local machine!

# ssh -n root@server 'tail -f /var/log/apache2/access.log' | goaccess -

Note: SSH requires -n so GoAccess can read from stdin. Also, make sure to use SSH keys for authentication as it won't work if a passphrase is required.

Troubleshooting

We receive many questions and issues that have been answered previously.

• Date/time matching problems? Check that your log format and the system locale in which you run GoAccess match. See #1571
• Problems with pattern matching? Spaces are often a problem, see for instance #136, #1579
• Other issues matching log entries: See >200 closed issues regarding log/date/time formats
• Problems with log processing? See >111 issues regarding log processing

Incremental log processing

GoAccess has the ability to process logs incrementally through its internal storage and dump its data to disk. It works in the following way:

1. A dataset must be persisted first with --persist, then the same dataset can be loaded with.
2. --restore. If new data is passed (piped or through a log file), it will append it to the original dataset.

NOTES

GoAccess keeps track of inodes of all the files processed (assuming files will stay on the same partition), in addition, it extracts a snippet of data from the log along with the last line parsed of each file and the timestamp of the last line parsed. e.g., inode:29627417|line:20012|ts:20171231235059

First, it compares if the snippet matches the log being parsed, if it does, it assumes the log hasn't changed drastically, e.g., hasn't been truncated. If the inode does not match the current file, it parses all lines. If the current file matches the inode, it then reads the remaining lines and updates the count of lines parsed and the timestamp. As an extra precaution, it won't parse log lines with a timestamp ≤ than the one stored.

Piped data works based off the timestamp of the last line read. For instance, it will parse and discard all incoming entries until it finds a timestamp >= than the one stored.

Examples

// last month access log
# goaccess access.log.1 --persist

then, load it with

// append this month access log, and preserve new data
# goaccess access.log --restore --persist

To read persisted data only (without parsing new data)

# goaccess --restore

Contributing

Any help on GoAccess is welcome. The most helpful way is to try it out and give feedback. Feel free to use the Github issue tracker and pull requests to discuss and submit code changes.

You can contribute to our translations by editing the .po files direct on Github or using the visual interface inlang.com

Enjoy!

Download Details:

Author: allinurl
Source Code: https://github.com/allinurl/goaccess 
License: MIT license

#c #nginx #cli #privacy #realtime #terminal 

Best 3 Predictions for Open Source in Confidential Computing

Best 3 Predictions for Open Source in Confidential Computing

Confidential computing is becoming more widely known by security and developer communities. Look out for these key trends in 2023.

It's a new year, which means it's time to predict what the next year will bring regarding future tech trends. After guessing the World Cup champion, I feel confident sharing my personal perspective on the confidential computing market in the near future.

What is confidential computing?

Confidential computing is the practice of isolating sensitive data and the techniques used to process it. This is as important on your laptop, where your data must be isolated from other applications, as it is on the cloud, where your data must be isolated from thousands of other containers and user accounts. As you can imagine, open source is a significant component for ensuring that what you believe is confidential is actually confidential. This is because security teams can audit the code of an open source project.

Confidential computing is a big space. When I talk about confidential computing, I first think of workloads running inside trusted execution environments (TEE). There are several categories of such workloads:

• Off-the-shelf products provided by a vendor
• Products built by a third party that need to be adapted and integrated into the customer environment
• Applications built and run by companies in support of their business

Off-the-shelf security products

Applications in this category already exist, and are expected to mature over the course of the year. The number of these applications is also expected to grow. Examples of applications include hardware security modules (HSM), security vaults, encryption services, and other security-related applications that render themselves to be the first choice for adopting confidential computing. While these applications exist, they constitute a fraction of the potential workloads that can run inside a TEE.

Third-party enablement applications

Workloads in this category are the ones built by software vendors for other customers. They require adaptation and integration for use. A vendor who makes this kind of software isn't a security vendor, but instead relies on security vendors (like Profian) to help them adapt their solutions to confidential computing. Such software includes AI software trained on customer data, or a database holding customer data for secure processing.

Homemade applications

These applications are built by customers for their internal use, leveraging assistance and enablement from confidential computing vendors.

Developing confidential computing technology

I suspect that third-party and homemade applications have similar dynamics. However, I expect more progress in a third-party enablement application segment, and here is why.

In the past year, a lot of discovery and educational activities were developed. Confidential computing is now better known, but it has yet to become a mainstream technology. The security and developer communities are gaining a better understanding of confidential computing and its benefits. If this discovery trend continues this year, it can influence more outlets, like conferences, magazines, and publications. This shows that these entities recognize the value of confidential computing. In time, they may start to offer more airtime for talks and articles on the subject.

Prediction #1: Pilot programs

The next phase after discovery is creating a pilot. Profian is seeing more interest among different vendors to move forward in building solutions and products that consciously target execution within trusted environments. This year, I expect to see a lot of pilot programs. Some of them can become production ready within the year. And some can pave the way for production-ready implementation next year.

Further interest is generated by greater visibility of confidential computing, a better understanding of the technology, and its value. In addition, the success of pilots, actual products, and services based on confidential computing platforms is guaranteed to generate interest.

Over the years, companies have collected and stored a lot of data about their business. If used using analytics and AI, this data helps companies improve business operations. They can also offer new or improved services and products to customers. Some of the data and models are valuable and need to be handled with security in mind. That's an ideal use case for confidential computing.

Companies looking to put their data to good use should start asking questions about security. This eventually leads them to discover confidential computing. From there, they can express interest in leveraging trusted environments to do computation. This, in turn, grows the attention of the companies (in the third-party category above) that provide products in this space to consider putting some of their products and offerings into confidential computing. I don't expect to see drastic changes in this area during this year. I do anticipate a shift in mindset toward recognizing the value of confidential computing and how it can help on a greater scale.

Prediction #2: Hardware and confidential computing

This year, I expect new hardware chips supporting confidential computing from different vendors and architectures. The hardware ecosystem is growing and that should continue this year. This gives more options to consumers, but also creates more requirements for hardware-agnostic solutions.

Prediction #3: Open standards

Finally, multiple security vendors are working on different deployment and attestation solutions. As those solutions mature, the need for some kind of interoperability is expected. Efforts for standardization are underway. But this year is likely to bring more pressure for projects to agree upon standardization and rules for interoperability.

Open source in confidential computing

Open source is key in confidential computing. The Enarx project provides a runtime environment, based on WebAssembly. This allows deploying a workload into a TEE in an architecture- and language-indifferent way. With the general awareness trends I've described above, I expect more engineers to join the open source ecosystem of confidential computing projects. This year, more developers might contribute to all elements of the stack, including the kernel, WebAssembly, Rust crates and tools, and Enarx itself.

Maybe one of those developers is you. If so, I look forward to collaborating with you.

Original article source at: https://opensource.com/

#opensource #computing #security #privacy 

The Community Building Open & Collaborative AI Anywhere at Any Scale

FedML: The Community Building Open and Collaborative AI Anywhere at Any Scale

FedML

At the current stage, FedML library provides a research and production integrated edge-cloud platform for Federated/Distributed Machine Learning at anywhere at any scale.

FedML Feature Overview

FedML logo reflects the mission of FedML Inc. FedML aims to build simple and versatile APIs for machine learning running anywhere at any scale. In other words, FedML supports both federated learning for data silos and distributed training for acceleration with MLOps and Open Source support, covering cutting-edge academia research and industrial grade use cases.

FedML Cheetah - Accelerate Model Training with User-friendly Distributed Training

FedML Parrot - Simulating federated learning in the real world (1) simulate FL using a single process (2) MPI-based FL Simulator (3) NCCL-based FL Simulator (fastest)

FedML Octopus - Cross-silo Federated Learning for cross-organization/account training, including Python-based edge SDK.

FedML Beehive - Cross-device Federated Learning for Smartphones and IoTs, including edge SDK for Android/iOS and embedded Linux.

FedML MLOps: FedML's machine learning operation pipeline for AI running anywhere at any scale.

Model Serving: we focus on providing a better user experience for edge AI.

Quick Start for Open Source Library

https://github.com/FedML-AI/FedML

Installation

To get started, let's install FedML first.

pip install fedml

For more installation methods, please refer to installing FedML.

A Quick Overview of the Code Architecture

In general, FedML source code architecture follows the paper which won Best Paper Award at NeurIPS 2020 (FL workshop). Its original idea is presented at the live video and white paper by FedML co-founder Dr. Chaoyang He.

After March 2022, FedML has been upgraded as an AI company which aims to provide machine learning capability anywhere at any scale. Now the python version of FedML https://github.com/FedML-AI/FedML- /tree/master/python is reorganized as follows:

core: The FedML low-level API package. This package implements distributed computing by communication backend like MPI, NCCL, MQTT, gRPC, PyTorch RPC, and also supports topology management. Other low-level APIs related to security and privacy are also supported. All algorithms and Scenarios are built based on the "core" package.

data: FedML will provide some default datasets for users to get started. Customization templates are also provided.

model: FedML model zoo.

device: FedML computing resource management.

simulation: FedML parrot can support (1) simulating FL using a single process (2) MPI-based FL Simulator (3) NCCL-based FL Simulator (fastest)

cross_silo: Cross-silo Federated Learning for cross-organization/account training

cross_device: Cross-device Federated Learning for Smartphones and IoTs

distributed: Distributed Training: Accelerate Model Training with Lightweight Cheetah

serve: Model serving, tailored for edge inference

mlops: APIs related to machine learning operation platform (open.fedml.ai)

centralized: Some centralized trainer code examples for benchmarking purposes.

utils: Common utilities shared by other modules.

Simplified APIs

Our philosophy of API design is to reduce the number of APIs to as few as possible while simultaneously keeping the flexibility.

For Simplicity, FedML Parrot has only one line API as the following example with fedml_config.yaml (an example is here: https://github.com/FedML-AI/FedML/blob/master/python/examples/simulation/sp_fedavg_mnist_lr_example/fedml_config.yaml)

# main.py

import fedml

if __name__ == "__main__":
    fedml.run_simulation()

python main.py

You will get the following output:

[FedML-Server(0) @device-id-0] [Sun, 01 May 2022 14:59:28] [INFO] [__init__.py:30:init] args = {'yaml_config_file': '', 'run_id': '0', 'rank': 0, 'yaml_paths': ['/Users/chaoyanghe/opt/anaconda3/envs/mnn37/lib/python3.7/site-packages/fedml-0.7.8-py3.7.egg/fedml/config/simulation_sp/fedml_config.yaml'], 'training_type': 'simulation', 'using_mlops': False, 'random_seed': 0, 'dataset': 'mnist', 'data_cache_dir': './data/mnist', 'partition_method': 'hetero', 'partition_alpha': 0.5, 'model': 'lr', 'federated_optimizer': 'FedAvg', 'client_id_list': '[]', 'client_num_in_total': 1000, 'client_num_per_round': 10, 'comm_round': 200, 'epochs': 1, 'batch_size': 10, 'client_optimizer': 'sgd', 'learning_rate': 0.03, 'weight_decay': 0.001, 'frequency_of_the_test': 5, 'using_gpu': False, 'gpu_id': 0, 'backend': 'single_process', 'log_file_dir': './log', 'enable_wandb': False}
[FedML-Server(0) @device-id-0] [Sun, 01 May 2022 14:59:28] [INFO] [device.py:14:get_device] device = cpu
[FedML-Server(0) @device-id-0] [Sun, 01 May 2022 14:59:28] [INFO] [data_loader.py:22:download_mnist] ./data/mnist/MNIST.zip
[FedML-Server(0) @device-id-0] [Sun, 01 May 2022 14:59:31] [INFO] [data_loader.py:57:load_synthetic_data] load_data. dataset_name = mnist
...

You can also customize the hyper-parameters with fedml_config.yaml. Check this tutorial for one-line example for details.

For flexibility, one-line API can also be expanded into five lines of APIs. To illustrate this, now let's switch to FedML Octopus (cross-silo federated learning) as example (Source code: https://github.com/FedML-AI/FedML/tree/master/python/examples/cross_silo/mqtt_s3_fedavg_mnist_lr_example).

In this example, the FL Client APIs are as follows:

import fedml
from fedml.cross_silo import Client

if __name__ == "__main__":
    args = fedml.init()

    # init device
    device = fedml.device.get_device(args)

    # load data
    dataset, output_dim = fedml.data.load(args)

    # load model
    model = fedml.model.create(args, output_dim)

    # start training
    client = Client(args, device, dataset, model)
    client.run()

With these APIs, you only need to tune the hyper-parameters with the configuration file fedml_config.yaml. An example is as follows:

common_args:
  training_type: "cross_silo"
  random_seed: 0

data_args:
  dataset: "mnist"
  data_cache_dir: "./../../../data"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  model_file_cache_folder: "./model_file_cache" # will be filled by the server automatically
  global_model_file_path: "./model_file_cache/global_model.pt"

train_args:
  federated_optimizer: "FedAvg"
  client_id_list: 
  client_num_in_total: 1000
  client_num_per_round: 2
  comm_round: 50
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03
  weight_decay: 0.001

validation_args:
  frequency_of_the_test: 5

device_args:
  worker_num: 2
  using_gpu: false
  gpu_mapping_file: config/gpu_mapping.yaml
  gpu_mapping_key: mapping_default

comm_args:
  backend: "MQTT_S3"
  mqtt_config_path: config/mqtt_config.yaml
  s3_config_path: config/s3_config.yaml

tracking_args:
  log_file_dir: ./log
  enable_wandb: false

Now let's run some examples as follows to get a sense of how FedML simplifies federated learning in diverse real-world settings.

FedML Parrot Examples

Simulation with a Single Process (Standalone):

• sp_fedavg_mnist_lr_example: Simulating FL using a single process in your personal laptop or server. This is helpful when researchers hope to try a quick algorithmic idea in small synthetic datasets (MNIST, shakespeare, etc.) and small models (ResNet-18, Logistic Regression, etc.).

Simulation with Message Passing Interface (MPI):

• mpi_torch_fedavg_mnist_lr_example: MPI-based Federated Learning for cross-GPU/CPU servers.

Simulation with NCCL-based MPI (the fastest training):

• In case your cross-GPU bandwidth is high (e.g., InfiniBand, NVLink, EFA, etc.), we suggest using this NCCL-based MPI FL simulator to accelerate your development.

FedML Octopus Examples

Horizontal Federated Learning:

• mqtt_s3_fedavg_mnist_lr_example: an example to illustrate how to run horizontal federated learning in data silos (hospitals, banks, etc.)

Hierarchical Federated Learning:

• hierarchical_fedavg_mnist_lr_example: an example to illustrate how to run hierarchical federated learning in data silos (hospitals, banks, etc.). Here hierarchical means that inside each FL Client (data silo), there are multiple GPUs that can run local distributed training with PyTorch DDP, and then the FL server aggregates globally from the results received from all FL Clients.

FedML on Smartphone and IoTs

See the introduction and tutorial at FedML/android.

See the introduction and tutorial at FedML/iot

MLOps User Guide

https://open.fedml.ai

Currently, the project developed based on FedML Octopus (cross-silo) and Beehive (cross-device) can be smoothly deployed into the real-world system using FedML MLOps.

FedML MLOps Platform simplifies the workflow of federated learning anywhere at any scale. It enables zero-code, lightweight, cross-platform, and provably secure federated learning. It enables machine learning from decentralized data at various users/silos/edge nodes, without the need to centralize any data to the cloud, hence providing maximum privacy and efficiency.

The above figure shows the workflow. Such a workflow is handled by web UI without the need to handle complex deployment. Check the following live demo for details:

3 Minutes Introduction: https://www.youtube.com/watch?v=E1k05jd1Tyw

A detailed guidance for the MLOps can be found at FedML MLOps User Guide.

More Resource

Reference

@article{chaoyanghe2020fedml,
  Author = {He, Chaoyang and Li, Songze and So, Jinhyun and Zhang, Mi and Wang, Hongyi and Wang, Xiaoyang and Vepakomma, Praneeth and Singh, Abhishek and Qiu, Hang and Shen, Li and Zhao, Peilin and Kang, Yan and Liu, Yang and Raskar, Ramesh and Yang, Qiang and Annavaram, Murali and Avestimehr, Salman},
  Journal = {Advances in Neural Information Processing Systems, Best Paper Award at Federate Learning Workshop},
  Title = {FedML: A Research Library and Benchmark for Federated Machine Learning},
  Year = {2020}
}

Ecosystem

FedML Ecosystem facilitates federated learning research and productization in diverse application domains. With the foundational support from FedML Core Framework, it supports FedNLP (Natural Language Processing), FedCV (Computer Vision), FedGraphNN (Graph Neural Networks), and FedIoT (Internet of Things). Please read this guidance for details: https://doc.fedml.ai/starter/ecosystem.html.

Publication

FedML’s core technology is backed by years of cutting-edge research represented in 50+ publications in ML/FL Algorithms, Security/Privacy, Systems, and Applications.

1. Vision Paper for High Scientific Impacts
2. System for Large-scale Distributed/Federated Training
3. Training Algorithms for FL
4. Security/privacy for FL
5. AI Applications A Full-stack of Scientific Publications in ML Algorithms, Security/Privacy, Systems, Applications, and Visionary Impacts

Please check this publication list for details.

Video (Invited Talks)

Trustworthy and Scalable Federated Learning. Federated Learning One World Seminar (FLOW). By Salman Avestimehr

Distributed ML for Federated Learning feat. Chaoyang He. Stanford MLSys Seminar. By Chaoyang He

Contributed Talk for FedML Library. Best Paper Award at NeurIPS 2020 Federated Learning Workshop. By Chaoyang He

Code of Conduct

FedML embraces and thrive through open-source. We welcome all kinds of contributions from the community. Kudos to all of our amazing contributors!
FedML has adopted Contributor Covenant.

FedML Homepage: https://fedml.ai/ 
FedML Open Source: https://github.com/FedML-AI 
FedML Platform: https://open.fedml.ai 
FedML Use Cases: https://open.fedml.ai/platform/appStore 
FedML Documentation: https://doc.fedml.ai 
FedML Blog: https://medium.com/@FedML 
FedML Research: https://fedml.ai/research-papers/ 
FedML Product Overview: https://medium.com/@FedML/fedml-ai-platform-releases-the-worlds-federated-learning-open-platform-on-public-cloud-with-an-8024e68a70b6

Join the Community: 
Slack: https://join.slack.com/t/fedml/shared_invite/zt-havwx1ee-a1xfOUrATNfc9DFqU~r34w 
Discord: https://discord.gg/9xkW8ae6RV

News

[2022/08/01] (Product Introduction) FedML AI platform releases the world’s federated learning open platform on the public cloud with an in-depth introduction of products and technologies! Please visit this blog for details.

[2022/03/15] (Fundraising): FedML, Inc. has finished the 1st-round fundraising. We are backed by top VCs who focus on AI, SaaS, and Blockchain/Web3/Crypto from the Bay Area, California of the USA.

[2022/02/14] (Company): FedML is upgraded as a Delaware-registered C-Corp company. Our headquarter is in California, USA. The two co-founders are CEO Salman Avestimehr (https://www.avestimehr.com/) and CTO Chaoyang He (https://chaoyanghe.com). We welcome contributors anywhere in the world.

[2021/02/01] (Award): #NeurIPS 2020# FedML won Best Paper Award at NeurIPS Federated Learning workshop 2020

Careers

FedML is hiring! Come and join us!

Download Details:

Author: FedML-AI
Source Code: https://github.com/FedML-AI/FedML 
License: Apache-2.0 license

#machinelearning #marketplace #privacy #deeplearning #blockchain 

A Peer-to-peer Chat App That Is Serverless, Decentralized, & Ephemeral

Chitchatter

Chitchatter is a free (as in both price and freedom) communication tool. It is designed to be the simplest way to connect with others privately and securely. It is:

• Fully open source (licensed under GPL v2)
• Peer-to-peer
  • Whenever possible, otherwise Open Relay is used to ensure reliable peer connection
• Encrypted (via WebRTC)
• Serverless
  • Public WebTorrent servers are only used for initial peer handshake
• Ephemeral
  • Message content is never persisted to disk
• Decentralized
  • There is no API server. All that's required for Chitchatter to function is availability of GitHub for static assets and public WebTorrent and STUN/TURN relay servers for establishing peer communication.
• Self-hostable

Chitchatter uses the Create React App tool chain. The secure networking and streaming magic would not be possible without Trystero.

How to use it

Open https://chitchatter.im/ and join a room to start chatting with anyone else who is in the room. By default, room names are random UUIDs that are generated client-side. To privately communicate with someone, it is recommended to join one of these randomly-generated rooms and share the URL (via the "🔗" button at the top of the page) to whomever you wish to communicate with via a secure medium of your choosing (such as Burner Note or Yopass). Your user name will be presented to you, and it would be good to share that with who you will be chatting with beforehand so they know they're talking to you.

Features

• Multiple peers per room (limited only by the number of peer connections your browser supports).
• Public and private rooms
• Video and audio chatting
• Screen sharing
• File sharing
  • Powered by WebTorrent and StreamSaver.js to support unlimited file size transfers
• Markdown support via react-markdown.
  • Includes support for syntax highlighting of code.
• Conversation backfilling from peers when a new participant joins
• Multiline message support (hold Shift and press Enter).
• Dark and light themes

Anti-features

• Messages are never persisted to disk. When you leave a peer room, messages are cleared from memory and cannot be retrieved.
• Chitchatter is an entirely client-side communication app. It uses public WebTorrent servers to establish peer connections and STUN/TURN relay servers when direct peer-to-peer connections cannot be established, but there is no Chitchatter API server.
• No analytics, tracking, or telemetry.
• This is a community-driven and unfunded project that makes no money. The users come first and there are no corporate or financial interests.

Why another chat app?

There is no shortage of user-friendly chat apps available, but most of them rely on a central service to facilitate communication. It is difficult to trust these central services, as commercial interests and government pressure can compel service operators to work against the best interest of the users. Even when when user data is handled in good faith by service operators, the possibility remains that encrypted data held at rest may be decrypted against the user's will.

Chitchatter designs around these risks with web meshes. There is no central service operator that stores or potentially mishandles communication data. Some services are required to establish an initial connection between peers, but the app uses direct peer-to-peer communication as much as possible. Any services that are used by the app have no association with the Chitchatter project and are publicly available for all to use.

Use cases

Chitchatter offers a private and secure solution for:

• Organizing groups of people, such as unions or political movements
• Video chatting with friends and family across operating systems (such as Android and iOS)
• Sharing files without an intermediary service
• IT troublshooting via screen sharing
• Livestreaming
• Sharing sensitive information like passwords
• Much more!

Veracity

The core of Chitchatter's security model is the fact that it is fully open source. You are free (and encouraged) to fully audit the project source code and infrastructure. Not only is the source code available under the terms of the GPL, but all build logs are publicly accessible as well.

If you would like to verify that the app hosted at https://chitchatter.im/ is the one that is hosted on GitHub, you can use dig:

$ dig chitchatter.im

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> chitchatter.im
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61332
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;chitchatter.im.                        IN      A

;; ANSWER SECTION:
chitchatter.im.         231     IN      CNAME   jeremyckahn.github.io.
jeremyckahn.github.io.  231     IN      A       185.199.111.153
jeremyckahn.github.io.  231     IN      A       185.199.110.153
jeremyckahn.github.io.  231     IN      A       185.199.109.153
jeremyckahn.github.io.  231     IN      A       185.199.108.153

To examine the static assets that are served to end users, you can audit the gh-pages branch.

Project roadmap

See the full ticket backlog here.

Environments

• Production environment: https://chitchatter.im/
  • Mirror: https://chitchatter.vercel.app/ (note that peers cannot connect across domains)
• Staging: https://chitchatter-git-develop-jeremyckahn.vercel.app/

Available Scripts

In the project directory, you can run:

npm dev

Runs the entire stack (client + WebTorrent tracker) locally.

npm start

Runs the front end app in the development mode. Uses public WebTorrent trackers.
Open http://localhost:3000 to view it in your browser.

The page will reload when you make changes. You may also see any lint errors in the console.

npm test

Launches the test runner in the interactive watch mode. See the section about running tests for more information.

npm run build

Builds the app for production to the build folder. It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.

Self-hosting

Chitchatter is designed to be forked and self-hosted. If you would like to change pairing or relay server configuration, or you simply prefer to control your own builds and versions, just fork this repo and follow the steps below.

Necessary steps after forking

Assuming you are hosting Chitchatter on GitHub Pages:

1. Change the homepage property in package.json to whatever URL your Chitchatter instance will be hosted from. This will be something like https://github_user_or_org_name.github.io/chitchatter/.
2. Define a DEPLOY_KEY GitHub Action secret (at https://github.com/github_user_or_org_name/chitchatter/settings/secrets/actions). See the docs for peaceiris/actions-gh-pages for more information.

Deployment

On GitHub

When hosted on GitHub Pages and the configuration above has been done, the Production environment is updated when the remote main branch is updated.

On non-GitHub hosts

Build the app with npm run build, and then serve the build directory. Any static file serving solution should work provided it is using a secure context.

Runtime configuration

Explore the files in src/config to modify pairing and relay server configuration.

Troubleshooting

Peers won't connect

This could happen for a variety of reasons. The most likely of which is that one or more peers cannot connect directly and must use the configured STUN/TURN relay as a fallback. The standard relay (https://www.metered.ca/tools/openrelay/) does not guarantee uptime, so it may simply be unavailable for some time. There's not much to do other than wait until it becomes available again.

Firefox-specific

Per #36, check your about:config settings and ensure that media.peerconnection.enabled is enabled.

⚠️ Disclaimer

By using Chitchatter, you agree to accept full responsibility for your actions related to its use. Additionally, you agree not to hold any contributors to the Chitchatter project responsible for any result of your use of it. The developers of Chitchatter do not endorse illegal activity and provide the software strictly for the purposes of lawful use.

Download Details:

Author: jeremyckahn
Source Code: https://github.com/jeremyckahn/chitchatter 
License: GPL-2.0 license

#serverless #react #chat #privacy 

Focus-ios: Firefox Focus for iOS

Firefox Focus for iOS

Browse like no one’s watching. The new Firefox Focus automatically blocks a wide range of online trackers — from the moment you launch it to the second you leave it. Easily erase your history, passwords and cookies, so you won’t get followed by things like unwanted ads.

Download on the App Store.

Getting Involved

We encourage you to participate in this open source project. We love Pull Requests, Bug Reports, ideas, (security) code reviews or any kind of positive contribution. Please read the Community Participation Guidelines.

• Chat: See #focus-ios for general discussion
• Bugs: File a new bug • Existing bugs

If you're looking for a good way to get started contributing, check out some good first issues.

We also tag recommended bugs for contributions with help wanted.

Main Branch

This branch works with Xcode 13 and supports iOS 13.0 and newer.

Pull requests should be submitted with main as the base branch.

Build Instructions

• Install Xcode 13 Xcode developer tools from Apple.
• Clone the repository:

git clone https://github.com/mozilla-mobile/focus-ios.git

• Pull in the project dependencies:

cd focus-ios
./checkout.sh

• Open Blockzilla.xcodeproj in Xcode.
• Build the Focus scheme in Xcode.

Download Details:

Author: Mozilla-mobile
Source Code: https://github.com/mozilla-mobile/focus-ios 
License: MPL-2.0 license

#swift #ios #firefox #privacy #browser

What is end-to-end encryption? How does it work, and why do we need it

End-to-end encryption (E2EE) is a method of secure communication that prevents third parties from accessing data while it's transferred from one end system or device to another.
Read more: https://kaagazzsecure.com/

#business  #privacy #datasecurity #filesecurity #kaagazzsecure
 

Privacy and Security Focused Segment-alternative, in Golang and React

RudderStack

The Customer Data Platform for Developers

As the leading open source Customer Data Platform (CDP), RudderStack provides data pipelines that make it easy to collect data from every application, website and SaaS platform, then activate it in your warehouse and business tools.

With RudderStack, you can build customer data pipelines that connect your whole customer data stack and then make them smarter by triggering enrichment and activation in customer tools based on analysis in your data warehouse. It's easy-to-use SDKs and event source integrations, Cloud Extract integrations, transformations, and expansive library of destination and warehouse integrations makes building customer data pipelines for both event streaming and cloud-to-warehouse ELT simple.

Key features

Warehouse-first: RudderStack treats your data warehouse as a first class citizen among destinations, with advanced features and configurable, near real-time sync.

Developer-focused: RudderStack is built API-first. It integrates seamlessly with the tools that the developers already use and love.

High Availability: RudderStack comes with at least 99.99% uptime. We have built a sophisticated error handling and retry system that ensures that your data will be delivered even in the event of network partitions or destinations downtime.

Privacy and Security: You can collect and store your customer data without sending everything to a third-party vendor. With RudderStack, you get fine-grained control over what data to forward to which analytical tool.

Unlimited Events: Event volume-based pricing of most of the commercial systems is broken. With RudderStack, you are be able to collect as much data as possible without worrying about overrunning your event budgets.

Segment API-compatible: RudderStack is fully compatible with the Segment API. So you don't need to change your app if you are using Segment; just integrate the RudderStack SDKs into your app and your events will keep flowing to the destinations (including data warehouses) as before.

Production-ready: Companies like Mattermost, IFTTT, Torpedo, Grofers, 1mg, Nana, OnceHub, and dozens of large companies use RudderStack for collecting their events.

Seamless Integration: RudderStack currently supports integration with over 90 popular tool and warehouse destinations.

User-specified Transformation: RudderStack offers a powerful JavaScript-based event transformation framework which lets you enhance or transform your event data by combining it with your other internal data. Furthermore, as RudderStack runs inside your cloud or on-premise environment, you can easily access your production data to join with the event data.

Get started

The easiest way to experience RudderStack is to sign up for RudderStack Cloud Free - a completely free tier of RudderStack Cloud.

You can also set up RudderStack on your platform of choice with these two easy steps:

Step 1: Set up RudderStack

• Docker
• Kubernetes
• Developer machine setup

Note: If you are planning to use RudderStack in production, we STRONGLY recommend using our Kubernetes Helm charts. We update our Docker images with bug fixes much more frequently than our GitHub repo.

Step 2: Verify the installation

Once you have installed RudderStack, send test events to verify the setup.

Architecture

RudderStack is an independent, stand-alone system with a dependency only on the database (PostgreSQL). Its backend is written in Go with a rich UI written in React.js.

A high-level view of RudderStack’s architecture is shown below:

For more details on the various architectural components, refer to our documentation.

Contribute

We would love to see you contribute to RudderStack. Get more information on how to contribute here.

Download Details:

Author: rudderlabs
Source Code: https://github.com/rudderlabs/rudder-server 
License: AGPL-3.0 license

#go #golang #react #security #privacy 

Piwigo: Manage your photos with Piwigo

Piwigo

Manage your photo library. Piwigo is open source photo gallery software for the web. Designed for organisations, teams and individuals.

The piwigo.org website introduces you to Piwigo. You'll find a demo, forums, wiki and news.

Requirements

• A webserver (Apache or nginx recommended)
• PHP 5.3 or greater (7+ recommended)
• MySQL 5 or greater or MariaDB equivalent
• ImageMagick (recommended) or PHP GD

Quick start install

NetInstall

• Download the NetInstall script
• Transfer the script to your web space with any FTP client
• Open the script in you web browser (for example http://example.com/piwigo-netinstall.php) and follow the steps

More information

Manual

• Download the latest stable version and unzip it
• Transfer everything to your web space with any FTP client
• Open your website (for example http://example.com/piwigo) and follow the steps

More information

If you do not have your own server, consider the piwigo.com hosting solution.

Contributing

Piwigo is widely driven by its community; if you want to improve the code, fork this repo and submit your changes to the master branch. See our Contribution guide.

Download Details:

Author: Piwigo
Source Code: https://github.com/Piwigo/Piwigo 
License: GPL-2.0, Unknown licenses found

#php #opensource #cms #privacy 

A Permissioned Implementation Of Ethereum Supporting Data Privacy

GoQuorum

GoQuorum is an Ethereum-based distributed ledger protocol with transaction/contract privacy and new consensus mechanisms.

GoQuorum is a fork of go-ethereum and is updated in line with go-ethereum releases.

Key enhancements over go-ethereum:

• Privacy - GoQuorum supports private transactions and private contracts through public/private state separation, and utilises peer-to-peer encrypted message exchanges (see Tessera) for directed transfer of private data to network participants
• Alternative Consensus Mechanisms - with no need for POW/POS in a permissioned network, GoQuorum instead offers multiple consensus mechanisms that are more appropriate for consortium chains:
  • QBFT - Improved version of IBFT that is interoperable with Hyperledger Besu
  • Istanbul BFT - a PBFT-inspired consensus algorithm with transaction finality, by AMIS.
  • Clique POA Consensus - a default POA consensus algorithm bundled with Go Ethereum.
  • Raft-based Consensus - a consensus model for faster blocktimes, transaction finality, and on-demand block creation
• Peer Permissioning - node/peer permissioning, ensuring only known parties can join the network
• Account Management - GoQuorum introduced account plugins, which allows GoQuorum or clef to be extended with alternative methods of managing accounts including external vaults.
• Pluggable Architecture - allows adding additional features as plugins to the core geth, providing extensibility, flexibility, and distinct isolation of GoQuorum features.
• Higher Performance - GoQuorum offers significantly higher performance throughput than public geth

Architecture

The above diagram is very high-level overview of component architecture used by GoQuorum. For more in-depth discussion of the components and how they interact, please refer to lifecycle of a private transaction.

Quickstart

The easiest way to get started is to use * quorum-dev-quickstart - a command line tool that allows users to set up a development GoQuorum network on their local machine in less than 2 minutes.

GoQuorum Projects

Check out some of the interesting projects we are actively working on:

• quorum-remix-plugin: The GoQuorum plugin for Ethereum's Remix IDE adds support for creating and interacting with private contracts on a GoQuorum network.
• Cakeshop: An integrated development environment and SDK for GoQuorum
• quorum-examples: GoQuorum demonstration examples
• Quorum-Kubernetes: Deploy GoQuorum on Kubernetes
• we3js-quorum: Extends web3.js to support GoQuorum and Hyperledger Besu specific APIs
• Zero Knowledge on GoQuorum
  • ZSL on GoQuorum
  • Anonymous Zether

Official Docker Containers

The official docker containers can be found under https://hub.docker.com/u/quorumengineering/

Third Party Tools/Libraries

The following GoQuorum-related libraries/applications have been created by Third Parties and as such are not specifically endorsed by J.P. Morgan. A big thanks to the developers for improving the tooling around GoQuorum!

• Quorum Blockchain Explorer - a Blockchain Explorer for GoQuorum which supports viewing private transactions
• Quorum-Genesis - A simple CL utility for GoQuorum to help populate the genesis file with voters and makers
• Quorum Maker - a utility to create GoQuorum nodes
• ERC20 REST service - a GoQuorum-supported RESTful service for creating and managing ERC-20 tokens
• Nethereum Quorum - a .NET GoQuorum adapter
• web3j-quorum - an extension to the web3j Java library providing support for the GoQuorum API
• Apache Camel - an Apache Camel component providing support for the GoQuorum API using web3j library. Here is the artcile describing how to use Apache Camel with Ethereum and GoQuorum https://medium.com/@bibryam/enterprise-integration-for-ethereum-fa67a1577d43

Contributing

GoQuorum is built on open source and we invite you to contribute enhancements. Upon review you will be required to complete a Contributor License Agreement (CLA) before we are able to merge. If you have any questions about the contribution process, please feel free to send an email to info@goquorum.com. Please see the Contributors guide for more information about the process.

Reporting Security Bugs

Security is part of our commitment to our users. At GoQuorum we have a close relationship with the security community, we understand the realm, and encourage security researchers to become part of our mission of building secure reliable software. This section explains how to submit security bugs, and what to expect in return.

All security bugs in GoQuorum and its ecosystem (Tessera, Cakeshop, ..etc) should be reported by email to security-quorum@consensys.net. Please use the prefix [security] in your subject. This email is delivered to GoQuorum security team. Your email will be acknowledged, and you'll receive a more detailed response to your email as soon as possible indicating the next steps in handling your report. After the initial reply to your report, the security team will endeavor to keep you informed of the progress being made towards a fix and full announcement.

If you have not received a reply to your email or you have not heard from the security team please contact any team member through GoQuorum slack security channel. Please note that GoQuorum discord channels are public discussion forum. When escalating to this medium, please do not disclose the details of the issue. Simply state that you're trying to reach a member of the security team.

Responsible Disclosure Process

GoQuorum project uses the following responsible disclosure process:

• Once the security report is received it is assigned a primary handler. This person coordinates the fix and release process.
• The issue is confirmed and a list of affected software is determined.
• Code is audited to find any potential similar problems.
• If it is determined, in consultation with the submitter, that a CVE-ID is required, the primary handler will trigger the process.
• Fixes are applied to the public repository and a new release is issued.
• On the date that the fixes are applied, announcements are sent to Quorum-announce.
• At this point you would be able to disclose publicly your finding.

Note: This process can take some time. Every effort will be made to handle the security bug in as timely a manner as possible, however it's important that we follow the process described above to ensure that disclosures are handled consistently.

Receiving Security Updates

The best way to receive security announcements is to subscribe to the Quorum-announce mailing list/channel. Any messages pertaining to a security issue will be prefixed with [security].

Comments on This Policy If you have any suggestions to improve this policy, please send an email to info@goquorum.com for discussion.

Download Details:

Author: ConsenSys
Source Code: https://github.com/ConsenSys/quorum 
License: LGPL-3.0, GPL-3.0 licenses found

#go #golang #privacy #ethereum #blockchain 

Vuepress-plugin-ackee: Ackee analytics Plugin for VuePress

Ackee Analytics Plugin for VuePress

This project helps to integrate Ackee Analytics in your VuePress site. Ackee is a self-hosted, privacy-friendly alternative to Google Analytics.

Install

For installation instructions on how to get Ackee up and running check this.

To install the VuePress plugin run one of the following commands:

npm install vuepress-plugin-ackee

# or

yarn add vuepress-plugin-ackee

Configuration

Add the vuepress plugin in your site or theme config file. The configuration is done as part of the plugin-configuration. Your configuration file is usually located under .vuepress/config.js. Extend it with this line:

module.exports = {
  plugins: [
    'ackee': { trackerUrl: 'https://link-to-your-ackee-instance.project.com', siteId: '12ab12ab-12ab-12ab-12ab-12ab12ab12ab' }

    // other plugins
    // ...
  ]
}

You will get parameters required from your Ackee instance.

Please note: The URL in the configuration shouldn't include /_.js.

For more details on how to work with VuePress plugins check the official docs.

Download Details:

Author: Spekulatius
Source Code: https://github.com/spekulatius/vuepress-plugin-ackee 
License: MIT license

#javascript #vue #privacy #analytics 

Ackee-report: CLI tool To Generate Performance Reports Of Websites

ackee-report

CLI tool to generate performance reports of websites using the self-hosted analytics tool Ackee.

👋 Introduction

ackee-report lets you create monthly website performance reports using your Ackee analytics data and either send them via email, generate a RSS feed or output it to a JSON file. It uses Ackee's Graphql API and can be configured to generate multiple reports for different websites, data ranges and recipients.

🚀 Get started

Install ackee-report globally via npm:

npm install ackee-report -g

ackee-report@latest requires Ackee >=v3.1.1. Use ackee-report@v1.1.3 for older versions of Ackee

After that ackee-report is ready to be used 🎉

⚙️ Configuration

On the first run ackee-report will ask you to input a few values:

• ackee server / ACKEE_SERVER - the endpoint of your Ackee instance
• ackee token / ACKEE_TOKEN - a permanent Ackee token (can be used instead of username and password, more info)
• ackee username ACKEE_USERNAME - your Ackee username (more info)
• ackee password ACKEE_PASSWORD - your Ackee password (more info)
• email host / EMAIL_HOST - the domain of the email server (more info)
• email port / EMAIL_PORT - the port of the email server (more info)
• email username / EMAIL_USERNAME - the username to use with the email server (more info)
• email password / EMAIL_PASSWORD - the password to use with the email server (more info)
• email from / EMAIL_FROM - the from address to use (more info)

The configuration will be stored in your home directory under ~/.config/configstore/ackee-report.json and can be changed at any point.

Environment Variables

If you don't want to interact with ackee-report via the CLI interface, you can also specify each configuration option as an environment variable e.g. ACKEE_TOKEN=<token>

Ackee API authentication

ackee-report needs access to your Ackee instance via the API in order to get all the data it needs to generate the report. You can choose any of the two authentication methods below:

Username and password:

Enter your username and password you use to login to the Ackee UI on the first run of ackee-report or change it in the config file later.

ackee-report will then use them to create a temporary access token each time it runs and use it to query the Ackee API.

Permanent access token (recommended):

The recommended way of authenticating ackee-report is with a permanent access token (only available with Ackee v2.2.0 or later).

You will have to create one via the Ackee UI under Settings/Permanent Tokens, then click on New permanent token and give it a name (doesn't matter what).

Copy the permanent token id and enter it on the first run of ackee-report or add it to the config file under ackee.token later.

The same token will then be used each time ackee-report runs to query the Ackee API.

Email setup

If you want to send your report via email, you have to specify your email providers SMTP server and credentials, as well as the from option:

• Host - smtp.example.com
• Port - 465
• Username - username@example.com
• Password - password
• From - username@example.com or Ackee <username@example.com>

Note: For port 465 ackee-report will use TLS when connecting to your email server, on all other ports it will use STARTTLS (#44)

Common providers:

Gmail

If you use gmail to send emails, use these values:

• Host - smtp.gmail.com
• Port - 465
• Username - your gmail username (your email address)
• Password - your gmail password or if you have 2FA enabled, an "Application Specific password"

SendGrid

If you use SendGrid to send emails, use these values:

• Host - smtp.sendgrid.net
• Port - 465
• Username - apikey (everyone's username is apiKey)
• Password - your API Key (generate one here)

MailGun

If you use SendGrid to send emails, use these values:

• Host - smtp.mailgun.org
• Port - 465
• Username - postmaster@yourdomain.name
• Password - get your password here

📚 Usage

Usage: ackee-report <command> [options]

CLI tool to generate performance reports of websites using the self-hosted analytics tool Ackee.

Commands:
  email [options]             Generate report and send it via email
  json [options]              Query API for data and output it to JSON file
  html [options]              Generate report and output it to a HTML file
  rss|xml [options]           Generate report as a RSS feed
  domains|domain [titles...]  get domain id by title
  config                      output current config
  help [command]              display help for command

Options:
  General:
    -d, --domain <titles...>    specify domains by title
    -i, --id <ids...>           specify domains by id
    -r, --range <range>         specify data range (default: "month")
    -l, --limit <number>        limit number of list items (default: 3)
    -e, --events [type]         get event data (default: false)
    -v, --version               output the version number
    -h, --help                  display help for command

  Email:
    -t, --to <recipient...>     to whom the report should be sent

  RSS/JSON/HTML:
    -o, --output <file>         path to output file (default: "report.[xml/json/html]")

Example call:
  $ ackee-report email --domain example.com --to hello@example.com

If you want to send the report periodically, you have to setup a cron job which runs the command at a set interval (example below).

🛠️ Examples

Generate a report for one domain and send it via email

ackee-report email -d example.com -t hello@example.com

This will generate a report for the domain example.com and send it via email to hello@example.com.

Multiple domains and recipients

ackee-report email -d example.com example2.com -t hello@example.com hey@example2.com

Include events in report

ackee-report email -d all -t hello@example.com -e

Average event type

ackee-report email -d all -t hello@example.com -e avg

Custom range

You can specify the data range of the report with the -r/--range option:

ackee-report email -d example.com -t hello@example.com -r day

Available values: day/week/month/six_months.

Note: The total views/range value is calculated by counting the views of the last x days since the program ran. For the month range for example, it is not the number of views in the current month, but the number of views in the last 30 days.

Send the report periodically (cron)

To send a report periodically, for example every month at the 28th at 23:55 setup a cron job like this:

55 23 28 * * ackee-report email -d example.com -t hello@example.com >> /tmp/ackee-report.log 2>&1

Note: We use the 28th as running on the last day of the month is very complicated with cron and Ackee resets its data on the first day of each month.

Note: You may have to specify the actual path to ackee-report. In that case, replace ackee-report in the command above with the output of which ackee-report.

If you are not familiar with cron, here's a tutorial on how to get started.

To send multiple reports to different people, add them all as seperate cron jobs.

Generate RSS feed

You can generate a RSS feed/xml file instead of sending the report via email:

ackee-report rss -d example.com -o output.xml

Output the report to a JSON file

You can also save the report in a JSON file:

ackee-report json -d example.com -o output.json

Output the email report to a HTML file

You can also save the report, which is normally sent via email, directly to an HTML file:

ackee-report html -d example.com -o index.html

🖼️ Screenshot

💻 Development

Issues and PRs are very welcome!

Run yarn lint or npm run lint to use eslint.

Please check out the contributing guide before you start.

To see differences with previous versions refer to the CHANGELOG.

❔ About

This library is an extension to the awesome privacy focused analytics tool Ackee.

Ackee was developed by @electerious, if you want to support him and the development of Ackee visit the Donate section on the Ackee repository.

Download Details:

Author: BetaHuhn
Source Code: https://github.com/BetaHuhn/ackee-report 
License: MIT license

#javascript #report #cli #privacy #analytics 

Ackee: Node.js Based analytics tool for Those Who Care About Privacy

Ackee 

Self-hosted, Node.js based analytics tool for those who care about privacy. Ackee runs on your own server, analyzes the traffic of your websites and provides useful statistics in a minimal interface.

👋 Introduction

Ackee is a self-hosted analytics tool that cares about privacy. We believe that you don't need to track every aspect of your visitors. Ackee keeps tracked data anonymized to avoid that users are identifiable, while still providing helpful insights. It's the right tool for everyone who doesn't need a full-featured marketing analytics platform like Google Analytics or Matomo.

• Self-hosted: Ackee runs on your own server and is 100% open-source
• Modern technologies: Lightweight Node.js and MongoDB architecture
• Beautiful: Minimal and focused interface
• No cookies: No unique user tracking and therefore no required cookie message
• Events: Track button clicks, newsletter subscriptions and more
• GraphQL API: Fully documented GraphQL API that allows you to build new tools upon Ackee

🚀 Get started

Get Ackee up and running…

• …with Docker Compose
• …with Docker
• …with Helm
• …without Docker
• …with Netlify
• …with Vercel
• …with Heroku
• …with Qovery
• …with Render
• …with Railway
• …with Koyeb

And configure Ackee and your server correctly…

• …with environment variables
• …with SSL and HTTPS enabled
• …with CORS headers

Take a look at the FAQ if you have any questions left.

📚 Documentation

Documentation and guides are located in the /docs folder. Also take a look at the FAQ if you have any questions left.

API

Ackee features a GraphQL API that allows you to build custom tools upon Ackee. Everything you see in the UI is made from data delivered by the API.

Options

Ackee uses environment variables and supports .env files in the root of the project if you want to store all variables in one file. Options »

Miscellaneous

Donate

I am working hard on continuously developing and maintaining Ackee. Please consider making a donation to keep the project going strong and me motivated.

• Become a GitHub sponsor
• Donate via PayPal
• Buy me a coffee

Articles

• Quit Google Analytics, Self-hosted Gatsby Statistics with Ackee
• Getting Ackee up and running with Heroku 🇪🇸
• Why I Self-Host My Website Analytics

Related

• ackee-tracker - Transfer data to Ackee
• ackee-bitbar - Ackee stats in your macOS menu bar
• ackee-lighthouse - Send Lighthouse reports to Ackee
• ackee-report - CLI tool to generate performance reports
• gatsby-plugin-ackee-tracker - Gatsby plugin for Ackee
• Soapberry - WordPress plugin for Ackee
• Ackee-PHP - A PHP Class for Ackee
• use-ackee - Use Ackee in React
• nuxt-ackee - Nuxt.js module for Ackee
• ngx-ackee-wrapper - Angular wrapper for Ackee
• django-ackee-middleware - Django middleware for Ackee
• gridsome-plugin-ackee - Gridsome plugin for Ackee
• vuepress-plugin-ackee - VuePress plugin for Ackee
• svelte-ackee - Svelte module for Ackee
• ackee_dart - Ackee plugin for Dart/Flutter (pub.dev)
• ackee-tracker-consent - A consent banner to activate detailed tracking on Ackee

Links

• Follow Ackee on Twitter
• Vote for Ackee on ProductHunt

Download Details:

Author: Electerious
Source Code: https://github.com/electerious/Ackee 
License: MIT license

#javascript #privacy #analytics #node