Discover Devices Connected to The Local Network within Seconds

Landiscover

Landiscover is a command-line tool that allows to discover devices and services available in the local network.

Features:

• Discover devices and services within seconds
• Available for Linux, no external dependencies

This software combines multiple discovery techniques:

• Arping is used to find machines
• DNS protocol is used to find hostnames
• Multicast DNS (MDNS) is used to find machines and hostnames
• NetBIOS protocol is used to find machines and hostnames

Installation and usage

Install and run with Docker:

docker run --rm -it --network=host -e COLUMNS=$COLUMNS aler9/landiscover

Alternatively, you can download and run a precompiled binary from the release page.

Full command-line usage

usage: landiscover [<flags>] [<interface>]

landiscover v0.0.0

Machine and service discovery tool.

Flags:
  --help     Show context-sensitive help (also try --help-long and --help-man).
  --passive  do not send any packet

Args:
  [<interface>]  Interface to listen to

---

Download Details:

Author: aler9
Source Code: https://github.com/aler9/landiscover 
License: MIT license

#go #golang #service #discovery 

Marquez: Collect, Aggregate, and Visualize A Data Ecosystem's Metadata

Marquez  

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.

Status

Marquez is an LF AI & Data Foundation incubation project under active development, and we'd love your help!

Adopters

Want to be added? Send a pull request our way!

• Astronomer
• Datakin
• Northwestern Mutual

Try it!

Quickstart

Marquez provides a simple way to collect and view dataset, job, and run metadata using OpenLineage. The easiest way to get up and running is with Docker. From the base of the Marquez repository, run:

$ ./docker/up.sh

Tip: Use the --build flag to build images from source, and/or --seed to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our quickstart guide.

Note: Port 5000 is now reserved for MacOS. If running locally on MacOS, you can run ./docker/up.sh --api-port 9000 to configure the API to listen on port 9000 instead. Keep in mind that you will need to update the URLs below with the appropriate port number.

WEB UI

You can open http://localhost:3000 to begin exploring the Marquez Web UI. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run metadata of current and previous job runs, and much more!

HTTP API

The Marquez HTTP API listens on port 5000 for all calls and port 5001 for the admin interface. The admin interface exposes helpful endpoints like /healthcheck and /metrics. To verify the HTTP API server is running and listening on localhost, browse to http://localhost:5001. To begin collecting lineage metadata as OpenLineage events, use the LineageAPI or an OpenLineage integration.

Note: By default, the HTTP API does not require any form of authentication or authorization.

GRAPHQL

To explore metadata via graphql, browse to http://localhost:5000/graphql-playground. The graphql endpoint is currently in beta and is located at http://localhost:5000/api/v1-beta/graphql.

Documentation

We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under docs/.

Note: To begin collecting metadata with Marquez, follow our quickstart guide. Below you will find the steps to get up and running from source.

Modules

Marquez uses a multi-project structure and contains the following modules:

• api: core API used to collect metadata
• web: web UI used to view metadata
• clients: clients that implement the HTTP API
• chart: helm chart

Note: The integrations module was removed in 0.21.0, so please use an OpenLineage integration to collect lineage events easily.

Requirements

• Java 17
• PostgreSQL 12.1

Note: To connect to your running PostgreSQL instance, you will need the standard psql tool.

Building

To build the entire project run:

./gradlew build

The executable can be found under api/build/libs/

Configuration

To run Marquez, you will have to define marquez.yml. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.

Step 1: Create Database

When creating your database using createdb, we recommend calling it marquez:

$ createdb marquez

Step 2: Create marquez.yml

With your database created, you can now copy marquez.example.yml:

$ cp marquez.example.yml marquez.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.

By default, Marquez uses the following ports:

• TCP port 8080 is available for the HTTP API server.
• TCP port 8081 is available for the admin interface.

Note: All of the configuration settings in marquez.yml can be specified either in the configuration file or in an environment variable.

Running the HTTP API Server

$ ./gradlew :api:runShadow

Marquez listens on port 8080 for all API calls and port 8081 for the admin interface. To verify the HTTP API server is running and listening on localhost, browse to http://localhost:8081. We encourage you to familiarize yourself with the data model and APIs of Marquez. To run the web UI, please follow the steps outlined here.

Note: By default, the HTTP API does not require any form of authentication or authorization.

Related Projects

• OpenLineage: an open standard for metadata and lineage collection

Getting Involved

• Website: https://marquezproject.ai
• Source: https://github.com/MarquezProject/marquez
• Chat: https://marquezproject.slack.com
• Twitter: @MarquezProject

Contributing

See CONTRIBUTING.md for more details about how to contribute.

Reporting a Vulnerability

If you discover a vulnerability in the project, please open an issue and attach the "security" label.

---

Download Details:

Author: MarquezProject
Source Code: https://github.com/MarquezProject/marquez 
License: Apache-2.0 license

#machinelearning #metadata #data #discovery 

Intro to Advanced Data Discovery Benefits and its Features

Introduction to Advance Data Discovery

Advanced data discovery is not limited to data scientists or IT staff only in today's world. Even business users also demand it. Business users demand quick and easy preparation and analysis data, visualize and explore data, notate and highlight the data, and share the data with others to identify the important nuggets. Without advanced analytics, it is impossible to achieve this within seconds. But with the concept of advanced data discovery allows business users to leverage advanced analytics, which helps in the rapid return of investment, increases revenue, and lowers the total cost of ownership. The key to data democratization and data literacy is Augmented Analytics. When an advanced analytics application for enterprise customers is developed, it encourages team members to use advanced analytics and lets the organization grow Citizen Data Scientists.

What is Advanced Data Discovery?

• Advanced-Data Exploration helps enterprise users effectively prepare and view, analyze and discover information, note, highlight, and share data with others.
• Market users may use Sophisticated Data Analysis to discover the critical 'nuggets' hidden in traditional data, link the dots, detect exceptions, recognize patterns and trends, and help forecast performance.
• The best-advanced data discovery platform is intended for enterprise users with average skills to do all of this without technical experience, knowledge of mathematical science, or assistance from IT or trained data scientists.
• A platform for data exploration is a critical tool for any enterprise customer in your organization. With so many data sources, the consumers can’t know whether they have access to complete, accurate data for their organization to make decisions in so many places.

Importance of Analytics for Advanced Data Discovery

• With the correct Advanced Analytics Software, market users can access data integrated from multiple data sources. They can use the data for Advanced Data Discovery to gain insight into problems and opportunities, share information with other users, and be more efficient, motivated, and accountable.
• Advanced Analytics requires the detection, interpretation, and communication of meaningful patterns of information and considers and applies trends and patterns to make clear, fact-based choices.
• In other words, advanced analytics connects information to actions and strategies and allows the organization to set targets and objectives that are practical and feasible in terms of competition.
• Although in the past few years, businesses have turned to IT and data analysts to identify, evaluate and understand data.
• Today the business market is evolving too fast to wait for this information, but the truth is that business consumers need this information and expertise to do their job.

How Advanced-Data Discovery helps the organization in achieving its goals?

• Concepts such as Advanced Data Discovery and Augmented Analytics can seem elusive and daunting to the average enterprise. Nothing more from the truth can be there! The solutions available today for Advanced Analytics Applications are diverse and flexible. The right intelligent technology exploration strategy will promote data democratization, social BI, and enthusiastic consumer acceptance around the organization at every stage of the business.
• The required Advanced Data Discovery helps business users leverage complex analytics in an elegant, easy-to-use environment. It turns business users with average technical expertise into Citizen Data Scientists.
• These data exploration tools deliver precise, concise results that allow the enterprise in any division and place to rapidly and easily prepare and analyze information and model and explore it, notice and highlight data, and exchange data around the company.
• Advanced analysis of data is not out of reach for our squad. The correct Advanced Analytics Platform helps any consumer perform research without technical experience, knowledge of predictive analysis, or assistance from IT or professional data scientists.

What are the features of Advanced Data Discovery?

The common features of Advanced Data Discovery are listed below:

Preparation of Self-Serve Data

Enables enterprise users to perform sophisticated data analysis and auto-suggest partnerships, demonstrates the importance and significance of critical variables, proposes data type casts, data consistency changes, and more.

Smart Visualization

Smart Data Visualization proposes the best choices for a given array or class of data to be visualized and plotted based on the nature, dimension, and form of data.

Predictive Analysis Plug n 'Play

Supported predictive modeling and predictive algorithms (associative, decision trees, sorting, clustering, and other techniques) allow market users to use Sophisticated Data Exploration and early prototyping recommendations to explore hypotheses and conclusions and minimize computational and experimental time and cost significantly. It empowers market customers with access to meaningful data to test theories and concepts without the aid of data scientists or IT staff.

Conclusion

It's quick to grasp the benefits of auto-suggestion and auto-recommendation. In the past few years, suppose market customers can use methods that complement average capabilities without needing advanced technical or analytical experience and information. In that case, they are more likely to use these services to obtain practical insight and make confident judgments and predictions.

• Discover here about Responsible Metaverse Characteristics
• Read how to Scale up and Govern AI initiatives

Original article source at: https://www.xenonstack.com/

#features #data #discovery #benefits 

Diversity Recommendation Systems in Machine Learning and AI

Personalization & Discovery that Does Social Good and Increases Customer Lifetime Value

Compiled from Google Images

Every day you are being influenced by machine learning and AI recommendation algorithms.

What you consume on social media through Facebook, Twitter, Instagram, the personalization you experience when you search, listen, and watch through Google, Spotify, Youtube, what you discover using Airbnb and UberEats, all of these products are powered by machine learning recommender systems.

Recommender systems influence our everyday lives

80% of all content consumed on Netflix and $98 billion of annual revenue on Amazon is driven by recommendation systems and these companies continue investing millions in building better versions of these algorithms.

There are two main types of recommender systems:

1. Collaborative filtering: finding similar users to you and recommending you something based on what that similar user liked.
2. Content-based filtering: taking your past history and behavior to make recommendations.

There is also a hybrid based recommender system, which mixes collaborative and content-based filtering. These machine learning and AI algorithms are what power the consumer products we use every day.

How recommender systems work

The problem is these algorithms are fundamentally optimizing for the same thing: similarities.

Recommendation algorithms make optimizations based on the key assumption that only similarities are good. If you like fantasy books, you will get recommended more fantasy books, if you like progressive politics, you will get recommended more progressive politics. Following these algorithms limit our world view and we fail to see new, interesting, and unique perspectives.

Recommender systems lead us down a one-track mind

Like a horse running with blinders, we fall into an echo chamber and the dangerous AI feedback loop where the algorithm’s outputs are reused to train new versions of the model. This narrows our thinking and reinforces biases. Recent events like the Facebook–Cambridge Analytica data breach demonstrate technology’s influence over human behavior and its impact on individuals and society.

Psychology and sociology agree: we fear what we do not know. When people become myopic, that is when the “us vs them” mentality is created and where prejudice is rooted. The civil unrest in the United States and around the world can be linked back to these concepts. Fortunately, research also demonstrates that diversity of perspectives creates understanding and connectedness.

#recommendation-system #discovery #artificial-intelligence #personalization #ai

Diversity Recommendation Systems in Machine Learning and AI

Every day you are being influenced by machine learning and AI recommendation algorithms.

What you consume on social media through Facebook, Twitter, Instagram, the personalization you experience when you search, listen, and watch through Google, Spotify, Youtube, what you discover using Airbnb and UberEats, all of these products are powered by machine learning recommender systems.

Recommender systems influence our everyday lives

80% of all content consumed on Netflix and $98 billion of annual revenue on Amazon is driven by recommendation systems and these companies continue investing millions in building better versions of these algorithms.

There are two main types of recommender systems:

1. Collaborative filtering: finding similar users to you and recommending you something based on what that similar user liked.
2. Content-based filtering: taking your past history and behavior to make recommendations.

There is also a hybrid based recommender system, which mixes collaborative and content-based filtering. These machine learning and AI algorithms are what power the consumer products we use every day.

How recommender systems work

The problem is these algorithms are fundamentally optimizing for the same thing: similarities.

Recommendation algorithms make optimizations based on the key assumption that only similarities are good. If you like fantasy books, you will get recommended more fantasy books, if you like progressive politics, you will get recommended more progressive politics. Following these algorithms limit our world view and we fail to see new, interesting, and unique perspectives.

Recommender systems lead us down a one-track mind

Like a horse running with blinders, we fall into an echo chamber and the dangerous AI feedback loop where the algorithm’s outputs are reused to train new versions of the model. This narrows our thinking and reinforces biases. Recent events like the Facebook–Cambridge Analytica data breach demonstrate technology’s influence over human behavior and its impact on individuals and society.

Psychology and sociology agree: we fear what we do not know. When people become myopic, that is when the “us vs them” mentality is created and where prejudice is rooted. The civil unrest in the United States and around the world can be linked back to these concepts. Fortunately, research also demonstrates that diversity of perspectives creates understanding and connectedness.

#recommendation-system #discovery #artificial-intelligence #machine-learning #personalization

The Search for Originality

Science has an impressive track record of originality. It may seem like no new philosophical ideas are being discovered, but that’s not true — they’re just comparatively smaller discoveries than those in the sciences.
This may be due to the fact that we are much more blind in the sciences than we are in philosophy. I suspect that’s because the entire web of possible knowledge in philosophy is likely smaller than the totality of possible knowledge in the natural sciences. Additionally, humans have only been doing good science for a few centuries, whereas we’ve been doing philosophy for millennia. Also, science is hard and we’re dumbasses.
But just because most of us don’t make original contributions to science or philosophy doesn’t mean we aren’t creating anything original.
As a species, we’re constantly developing new tools, languages, modes of thought, inventions, and most importantly, art.
Art is our crowning evolutionary achievement as far as I’m concerned.
In art, originality is easy to come by.
While there are strict rules in science and philosophy, there are no rules in art.
I could record a random tune in 30 seconds and it could be completely original. In fact, I just did!
But even though originality may be much more easily achieved in the arts, it’s rare to come up with an entirely new style or genre. People who do this are called pioneers, and they are rightly celebrated.
For example, Martha Graham pioneered an entirely new school of movement: modern dance. There were free-form dancers before her, but she transformed the conception of dance itself. She created an entirely new section on the tapestry of artistic movement. Before her, we believed the tapestry to be only so big, but she showed us all that it was wider than we had previously imagined.

#science #ideas #philosophy #discovery #psychology

User Interfaces for Data Science

Many models are let down by a poorly implemented user interface. The understanding the interface requirement is asking the user the right questions.

#data-science #user-interface #discovery #user-experience