Arno  Bradtke

Arno Bradtke

1598110560

Jaeger’s multitenancy with Elasticsearch

For the past year, due to the need of distributed transaction monitoring and root cause analysis in a complex distributed micro-service environment, we introduced Jaeger framework to help us tackle the problem. Since our platform is being used by multiple tenants, we had to take a decision on how we would implement the multi-tenancy Jaeger with Elasticsearch as backend.


This is a practical exercise on how to setup Jaeger with Elasticsearch to support multiple tenants. But first, you should read the following article Jaeger and multitenancy which talks about various multi-tenancy options with Jaeger.

The context

We are building and running a platform based on Kubernetes, which allows our customers to build and deploy their own applications using our platform, thus the specific requirements when it comes to tracing data:

  • One Elasticsearch instance supporting all the tenants,
  • Each tenant’s trace data has to be persisted separately — with various retention timeframes,
  • Ability for every tenant to view and query its own tracing data,
  • As minimal development activities as possible, thus reusing the existing Jaeger functionalities.

The solution

After going through enough material from different sources to have a clear picture, I decided on the following solution which consists of:

  • An Elasticsearch instance installed in an independent namespace of the tenants ones — we want to manage our own ES cluster,
  • Install a Jaeger collector for each tenant, configured to use the ES cluster with the specific tenant name/id,
  • Install the Jaeger agents as sidecars to the services that are being traced,
  • For each tenant configure it’s own Jaeger’s elastic search index cleaner settings,
  • Everything set up with 0 development effort.

#multitenancy #kubernetes #helm #jaeger #elasticsearch

What is GEEK

Buddha Community

Jaeger’s multitenancy with Elasticsearch
Mike  Kozey

Mike Kozey

1656151740

Test_cov_console: Flutter Console Coverage Test

Flutter Console Coverage Test

This small dart tools is used to generate Flutter Coverage Test report to console

How to install

Add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

dev_dependencies:
  test_cov_console: ^0.2.2

How to run

run the following command to make sure all flutter library is up-to-date

flutter pub get
Running "flutter pub get" in coverage...                            0.5s

run the following command to generate lcov.info on coverage directory

flutter test --coverage
00:02 +1: All tests passed!

run the tool to generate report from lcov.info

flutter pub run test_cov_console
---------------------------------------------|---------|---------|---------|-------------------|
File                                         |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/                                     |         |         |         |                   |
 print_cov.dart                              |  100.00 |  100.00 |   88.37 |...,149,205,206,207|
 print_cov_constants.dart                    |    0.00 |    0.00 |    0.00 |    no unit testing|
lib/                                         |         |         |         |                   |
 test_cov_console.dart                       |    0.00 |    0.00 |    0.00 |    no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
 All files with unit testing                 |  100.00 |  100.00 |   88.37 |                   |
---------------------------------------------|---------|---------|---------|-------------------|

Optional parameter

If not given a FILE, "coverage/lcov.info" will be used.
-f, --file=<FILE>                      The target lcov.info file to be reported
-e, --exclude=<STRING1,STRING2,...>    A list of contains string for files without unit testing
                                       to be excluded from report
-l, --line                             It will print Lines & Uncovered Lines only
                                       Branch & Functions coverage percentage will not be printed
-i, --ignore                           It will not print any file without unit testing
-m, --multi                            Report from multiple lcov.info files
-c, --csv                              Output to CSV file
-o, --output=<CSV-FILE>                Full path of output CSV file
                                       If not given, "coverage/test_cov_console.csv" will be used
-t, --total                            Print only the total coverage
                                       Note: it will ignore all other option (if any), except -m
-p, --pass=<MINIMUM>                   Print only the whether total coverage is passed MINIMUM value or not
                                       If the value >= MINIMUM, it will print PASSED, otherwise FAILED
                                       Note: it will ignore all other option (if any), except -m
-h, --help                             Show this help

example run the tool with parameters

flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock
---------------------------------------------|---------|---------|---------|-------------------|
File                                         |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/                                     |         |         |         |                   |
 print_cov.dart                              |  100.00 |  100.00 |   88.37 |...,149,205,206,207|
lib/                                         |         |         |         |                   |
 test_cov_console.dart                       |    0.00 |    0.00 |    0.00 |    no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
 All files with unit testing                 |  100.00 |  100.00 |   88.37 |                   |
---------------------------------------------|---------|---------|---------|-------------------|

report for multiple lcov.info files (-m, --multi)

It support to run for multiple lcov.info files with the followings directory structures:
1. No root module
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
2. With root module
<root>/coverage/lcov.info
<root>/lib/src
<root>/<module_a>
<root>/<module_a>/coverage/lcov.info
<root>/<module_a>/lib/src
<root>/<module_b>
<root>/<module_b>/coverage/lcov.info
<root>/<module_b>/lib/src
...
You must run test_cov_console on <root> dir, and the report would be grouped by module, here is
the sample output for directory structure 'with root module':
flutter pub run test_cov_console --file=coverage/lcov.info --exclude=_constants,_mock --multi
---------------------------------------------|---------|---------|---------|-------------------|
File                                         |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/                                     |         |         |         |                   |
 print_cov.dart                              |  100.00 |  100.00 |   88.37 |...,149,205,206,207|
lib/                                         |         |         |         |                   |
 test_cov_console.dart                       |    0.00 |    0.00 |    0.00 |    no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
 All files with unit testing                 |  100.00 |  100.00 |   88.37 |                   |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_a -                            |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/                                     |         |         |         |                   |
 print_cov.dart                              |  100.00 |  100.00 |   88.37 |...,149,205,206,207|
lib/                                         |         |         |         |                   |
 test_cov_console.dart                       |    0.00 |    0.00 |    0.00 |    no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
 All files with unit testing                 |  100.00 |  100.00 |   88.37 |                   |
---------------------------------------------|---------|---------|---------|-------------------|
---------------------------------------------|---------|---------|---------|-------------------|
File - module_b -                            |% Branch | % Funcs | % Lines | Uncovered Line #s |
---------------------------------------------|---------|---------|---------|-------------------|
lib/src/                                     |         |         |         |                   |
 print_cov.dart                              |  100.00 |  100.00 |   88.37 |...,149,205,206,207|
lib/                                         |         |         |         |                   |
 test_cov_console.dart                       |    0.00 |    0.00 |    0.00 |    no unit testing|
---------------------------------------------|---------|---------|---------|-------------------|
 All files with unit testing                 |  100.00 |  100.00 |   88.37 |                   |
---------------------------------------------|---------|---------|---------|-------------------|

Output to CSV file (-c, --csv, -o, --output)

flutter pub run test_cov_console -c --output=coverage/test_coverage.csv

#### sample CSV output file:
File,% Branch,% Funcs,% Lines,Uncovered Line #s
lib/,,,,
test_cov_console.dart,0.00,0.00,0.00,no unit testing
lib/src/,,,,
parser.dart,100.00,100.00,97.22,"97"
parser_constants.dart,100.00,100.00,100.00,""
print_cov.dart,100.00,100.00,82.91,"29,49,51,52,171,174,177,180,183,184,185,186,187,188,279,324,325,387,388,389,390,391,392,393,394,395,398"
print_cov_constants.dart,0.00,0.00,0.00,no unit testing
All files with unit testing,100.00,100.00,86.07,""

Installing

Use this package as an executable

Install it

You can install the package from the command line:

dart pub global activate test_cov_console

Use it

The package has the following executables:

$ test_cov_console

Use this package as a library

Depend on it

Run this command:

With Dart:

 $ dart pub add test_cov_console

With Flutter:

 $ flutter pub add test_cov_console

This will add a line like this to your package's pubspec.yaml (and run an implicit dart pub get):

dependencies:
  test_cov_console: ^0.2.2

Alternatively, your editor might support dart pub get or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:test_cov_console/test_cov_console.dart';

example/lib/main.dart

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
        // This makes the visual density adapt to the platform that you run
        // the app on. For desktop platforms, the controls will be smaller and
        // closer together (more dense) than on mobile platforms.
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key, required this.title}) : super(key: key);

  // This widget is the home page of your application. It is stateful, meaning
  // that it has a State object (defined below) that contains fields that affect
  // how it looks.

  // This class is the configuration for the state. It holds the values (in this
  // case the title) provided by the parent (in this case the App widget) and
  // used by the build method of the State. Fields in a Widget subclass are
  // always marked "final".

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  int _counter = 0;

  void _incrementCounter() {
    setState(() {
      // This call to setState tells the Flutter framework that something has
      // changed in this State, which causes it to rerun the build method below
      // so that the display can reflect the updated values. If we changed
      // _counter without calling setState(), then the build method would not be
      // called again, and so nothing would appear to happen.
      _counter++;
    });
  }

  @override
  Widget build(BuildContext context) {
    // This method is rerun every time setState is called, for instance as done
    // by the _incrementCounter method above.
    //
    // The Flutter framework has been optimized to make rerunning build methods
    // fast, so that you can just rebuild anything that needs updating rather
    // than having to individually change instances of widgets.
    return Scaffold(
      appBar: AppBar(
        // Here we take the value from the MyHomePage object that was created by
        // the App.build method, and use it to set our appbar title.
        title: Text(widget.title),
      ),
      body: Center(
        // Center is a layout widget. It takes a single child and positions it
        // in the middle of the parent.
        child: Column(
          // Column is also a layout widget. It takes a list of children and
          // arranges them vertically. By default, it sizes itself to fit its
          // children horizontally, and tries to be as tall as its parent.
          //
          // Invoke "debug painting" (press "p" in the console, choose the
          // "Toggle Debug Paint" action from the Flutter Inspector in Android
          // Studio, or the "Toggle Debug Paint" command in Visual Studio Code)
          // to see the wireframe for each widget.
          //
          // Column has various properties to control how it sizes itself and
          // how it positions its children. Here we use mainAxisAlignment to
          // center the children vertically; the main axis here is the vertical
          // axis because Columns are vertical (the cross axis would be
          // horizontal).
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'You have pushed the button this many times:',
            ),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ), // This trailing comma makes auto-formatting nicer for build methods.
    );
  }
}

Author: DigitalKatalis
Source Code: https://github.com/DigitalKatalis/test_cov_console 
License: BSD-3-Clause license

#flutter #dart #test 

Arno  Bradtke

Arno Bradtke

1598110560

Jaeger’s multitenancy with Elasticsearch

For the past year, due to the need of distributed transaction monitoring and root cause analysis in a complex distributed micro-service environment, we introduced Jaeger framework to help us tackle the problem. Since our platform is being used by multiple tenants, we had to take a decision on how we would implement the multi-tenancy Jaeger with Elasticsearch as backend.


This is a practical exercise on how to setup Jaeger with Elasticsearch to support multiple tenants. But first, you should read the following article Jaeger and multitenancy which talks about various multi-tenancy options with Jaeger.

The context

We are building and running a platform based on Kubernetes, which allows our customers to build and deploy their own applications using our platform, thus the specific requirements when it comes to tracing data:

  • One Elasticsearch instance supporting all the tenants,
  • Each tenant’s trace data has to be persisted separately — with various retention timeframes,
  • Ability for every tenant to view and query its own tracing data,
  • As minimal development activities as possible, thus reusing the existing Jaeger functionalities.

The solution

After going through enough material from different sources to have a clear picture, I decided on the following solution which consists of:

  • An Elasticsearch instance installed in an independent namespace of the tenants ones — we want to manage our own ES cluster,
  • Install a Jaeger collector for each tenant, configured to use the ES cluster with the specific tenant name/id,
  • Install the Jaeger agents as sidecars to the services that are being traced,
  • For each tenant configure it’s own Jaeger’s elastic search index cleaner settings,
  • Everything set up with 0 development effort.

#multitenancy #kubernetes #helm #jaeger #elasticsearch

Rusty  Shanahan

Rusty Shanahan

1598155740

Elasticsearch 7.x Backup — “Snapshot & Restore” on AWS S3

In 2016 I wrote an Article about Elasticsearch Backup, it had and still has quite good interests from people. I decided to start a new series of articles with the Backup topic as the main argument.

The old article covered Snapshot & Restore functionalities based on Elasticsearch 2.4.x and the upcoming version, the 5.0. As it was 4 years ago I choose to refresh this tutorial and making it the first of a series of more.

I will prepare a small article on how to use the snapshot & restore functionality with different cloud-provider. This article is based on Elasticsearch 7.x, it doesn’t mean it couldn’t work on older versions but I focused on the latest one.

Elasticsearch Snapshot & Restore

Elasticsearch has a smart solution to backup single indices or entire clusters to remote shared filesystem or S3 or HDFS. The snapshot ES creates does not so resource consuming and is relatively small.

The idea behind these snapshots is that they are not “archive” in a strict sense, these snapshots can only be read by a version of Elasticsearch that is capable to read the index version stored inside the snapshot.

So you can follow this quick scheme if you want to restore ES snapshots :

  • A snapshot of an index created in 6.x can be restored to 7.x.
  • A snapshot of an index created in 5.x can be restored to 6.x.
  • A snapshot of an index created in 2.x can be restored to 5.x.
  • A snapshot of an index created in 1.x can be restored to 2.x.

Snapshots of indices created with ES 1.x cannot be restored to 5.x or 6.x, snapshots of indices created in 2.x cannot be restored to 6.x or 7.x, and snapshots of indices created in 5.x cannot be restored to 7.x or 8.x.

#elasticsearch-snapshot #elasticsearch-plugins #elasticsearch #backup #elasticsearch-backup #aws

Kole  Haag

Kole Haag

1602405154

Jaeger Persistent Storage with Elasticsearch, Cassandra and Kafka

Running systems in production involve requirements for high availability, resilience and recovery from failure. When running cloud-native applications this becomes even more critical, as the base assumption in such environments is that compute nodes will suffer outages, Kubernetes nodes will go down and microservices instances are likely to fail, yet the service is expected to remain up and running.

In a recent post, I presented the different Jaeger components and best practices for deploying Jaeger in production. In that post, I mentioned that Jaeger uses external services for ingesting and persisting the span data, such as Elasticsearch, Cassandra and Kafka. This is due to the fact that the Jaeger Collector is a stateless service and you need to point it to some sort of storage to which it will forward the span data.

In this post, I’d like to discuss how to ingest and persist Jaeger trace data in production to ensure resilience and high availability, and the external services you need to set up for that. I’ll cover:

  • Standard persistent storage for Jaeger with Elasticsearch and Cassandra
  • Alternative persistent storage with gRPC plugin
  • Handling high load tracing data streams with Kafka
  • Jaeger persistence during development with jaegertracing all-in-one

Deploying Jaeger with Elasticsearch, Kafka or other External Services

Jaeger deployments may involve additional services such as Elasticsearch, Cassandra and Kafka. But do these services come as part of Jaeger’s installation and how are these services deployed?

The Jaeger Operator and Jaeger’s Helm chart (see Jaeger’s deployment tools on this post) offer the option of a self-provisioned Elasticsearch/Cassandra/Kafka cluster (in which Jaeger deployment also deploys these clusters), as well as the option of connecting to an existing cluster.

The self-provisioned option offers a good starting point, but you may prefer to deploy these services independently for better flexibility and control over the way these clusters are deployed, managed, monitored, upgraded and secured, in accordance with your team’s DevOps practices.

#jaeger #elasticsearch #kafka #cassandra #observability #database #database-storage #distributed-tracing-system

Writing .Net Core Application’s Log into ElasticSearch with NLog

We could write all types of log into ElasticSearch by configuring our application.

As the first step, you need to create an empty web application(.net core) in Visual studio.

creating new project

Now right click on the project and select** Add new Item** and search config to find Web Configuration File option. Set nlog.config as the file name.

Add new Item and search config to find Web Configuration File option. Set nlog.config as the file name

Write the below code into the newly added config file.

XML

1

<?xml version="1.0" encoding="utf-8" ?>

2

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"

3

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

4

      autoReload="true"> 

5

  <extensions>

6

    <add assembly="NLog.Targets.ElasticSearch"/>

7

  </extensions>

8

9

10

  <targets async="true"> 

11

    <target name="elastic"

12

            xsi:type="ElasticSearch"

13

            index="YourAppName" uri="http://192.168.50.60:9200" 

14

            layout ="API:SpecificName |${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" >

15

    </target> 

16

  </targets>

17

18

  <rules>

19

    <!--All logs, including from Microsoft-->

20

    <!--<logger name="*" minlevel="Trace" writeTo="allfile" />

21

22

    <logger name="*" minlevel="Trace" writeTo="ownFile-web" />

23

24

    <logger name="*" minlevel="Trace" writeTo="database" />-->

25

26

    <logger name="*" minlevel="Trace" writeTo="elastic" />

27

28

  </rules>

29

</nlog>

Before set the logging config, we should install some related packages, For this purpose, select NugetPackageManager from Tools menu and after that select PackageManagerConsole. We could write all types of log into ElasticSearch by configuring our application.

As the first step, you need to create an empty web application(.net core) in Visual studio.

Now right click on the project and select Add new Item and search config to find Web Configuration File option. Set nlog.config as the file name.

Write the below code into the newly added config file.

XML

1

2
<nlog xmlns=“http://www.nlog-project.org/schemas/NLog.xsd
3
     xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance
4
     autoReload=“true”>
5
 
6
   
7
 
8

9

10
 
11
   <target name=“elastic”
12
           xsi:type=“ElasticSearch”
13
           index=“YourAppName” uri=“http://192.168.50.60:9200
14
           layout =“API:SpecificName |${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}” >
15
   
16
 
17

18
 
19
   
20
   <!–
21

22
   
23

24
   –>
25

26
   
27

28
 
29


Before set the logging config, we should install some related packages, For this purpose, select NugetPackageManager from Tools menu and after that select PackageManagerConsole.

#programming #elasticsearch #nlog #.net core #.net #elasticsearch