Goes: A Library to interact with Elasticsearch in Go!

Goes : a library to interact with ElasticSearch

Supported operations

• index creation
• index removal
• simple indexing (document)
• bulk indexing
• search
• get

Example

You will find examples in example_test.go

Note: If you are switching to OwnLocal/goes from belogik/goes you will want to point whatever dependency management system you are using at the v1.0.0 tag to maintain backward compatibility (or change your code to support any API changes which may exist). The master branch will contain whatever the latest released version is and the develop branch will contain the currently in-progress version. Either of these branches may contain backwards-incompatible changes from the v1.0.0 tag (which contains the code as it was when the code was forked from belogik/goes).

Author: OwnLocal
Source Code: https://github.com/OwnLocal/goes 
License: BSD-3-Clause License

#go #golang 

What is GEEK

Buddha Community

What's new in the go 1.15

Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.

As Go promise for maintaining backward compatibility. After upgrading to the latest Go 1.15 version, almost all existing Golang applications or programs continue to compile and run as older Golang version.

#go #golang #go 1.15 #go features #go improvement #go package #go new features

Top 8 Machine Learning Libraries In Go Language One Must Know

Created by Google researchers, Go is a popular open-source programming language. The language includes many intuitive features, including a garbage collector, cross-platform, efficient concurrency, among others.

According to the Stack Overflow Developer Survey 2020, Go language is not only the fifth most loved programming language but also fetches the programmers the third-highest salary among other languages.

Below here, we list down the top machine learning libraries in Go language.

#opinions #go language #google ml tools #machine learning libraries #ml libraries #ml libraries in go

Go-elasticsearch: The Official Go Client for Elasticsearch

go-elasticsearch

The official Go client for Elasticsearch.

Compatibility

Language clients are forward compatible; meaning that clients support communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

When using Go modules, include the version in the import path, and specify either an explicit version or a branch:

require github.com/elastic/go-elasticsearch/v8 v8.0.0
require github.com/elastic/go-elasticsearch/v7 7.17

It's possible to use multiple versions of the client in a single project:

// go.mod
github.com/elastic/go-elasticsearch/v7 v7.17.0
github.com/elastic/go-elasticsearch/v8 v8.0.0

// main.go
import (
  elasticsearch7 "github.com/elastic/go-elasticsearch/v7"
  elasticsearch8 "github.com/elastic/go-elasticsearch/v8"
)
// ...
es7, _ := elasticsearch7.NewDefaultClient()
es8, _ := elasticsearch8.NewDefaultClient()

The main branch of the client is compatible with the current master branch of Elasticsearch.

Installation

Add the package to your go.mod file:

require github.com/elastic/go-elasticsearch/v8 main

Or, clone the repository:

git clone --branch main https://github.com/elastic/go-elasticsearch.git $GOPATH/src/github.com/elastic/go-elasticsearch

A complete example:

mkdir my-elasticsearch-app && cd my-elasticsearch-app

cat > go.mod <<-END
  module my-elasticsearch-app

  require github.com/elastic/go-elasticsearch/v8 main
END

cat > main.go <<-END
  package main

  import (
    "log"

    "github.com/elastic/go-elasticsearch/v8"
  )

  func main() {
    es, _ := elasticsearch.NewDefaultClient()
    log.Println(elasticsearch.Version)
    log.Println(es.Info())
  }
END

go run main.go

Usage

The elasticsearch package ties together two separate packages for calling the Elasticsearch APIs and transferring data over HTTP: esapi and elastictransport, respectively.

Use the elasticsearch.NewDefaultClient() function to create the client with the default settings.

es, err := elasticsearch.NewDefaultClient()
if err != nil {
  log.Fatalf("Error creating the client: %s", err)
}

res, err := es.Info()
if err != nil {
  log.Fatalf("Error getting response: %s", err)
}

defer res.Body.Close()
log.Println(res)

// [200 OK] {
//   "name" : "node-1",
//   "cluster_name" : "go-elasticsearch"
// ...

NOTE: It is critical to both close the response body and to consume it, in order to re-use persistent TCP connections in the default HTTP transport. If you're not interested in the response body, call io.Copy(ioutil.Discard, res.Body).

When you export the ELASTICSEARCH_URL environment variable, it will be used to set the cluster endpoint(s). Separate multiple adresses by a comma.

To set the cluster endpoint(s) programatically, pass a configuration object to the elasticsearch.NewClient() function.

cfg := elasticsearch.Config{
  Addresses: []string{
    "http://localhost:9200",
    "http://localhost:9201",
  },
  // ...
}
es, err := elasticsearch.NewClient(cfg)

To set the username and password, include them in the endpoint URL, or use the corresponding configuration options.

cfg := elasticsearch.Config{
  // ...
  Username: "foo",
  Password: "bar",
}

To set a custom certificate authority used to sign the certificates of cluster nodes, use the CACert configuration option.

cert, _ := ioutil.ReadFile(*cacert)

cfg := elasticsearch.Config{
  // ...
  CACert: cert,
}

To configure other HTTP settings, pass an http.Transport object in the configuration object.

cfg := elasticsearch.Config{
  Transport: &http.Transport{
    MaxIdleConnsPerHost:   10,
    ResponseHeaderTimeout: time.Second,
    TLSClientConfig: &tls.Config{
      MinVersion: tls.VersionTLS12,
      // ...
    },
    // ...
  },
}

See the _examples/configuration.go and _examples/customization.go files for more examples of configuration and customization of the client. See the _examples/security for an example of a security configuration.

The following example demonstrates a more complex usage. It fetches the Elasticsearch version from the cluster, indexes a couple of documents concurrently, and prints the search results, using a lightweight wrapper around the response body.

// $ go run _examples/main.go

package main

import (
  "bytes"
  "context"
  "encoding/json"
  "log"
  "strconv"
  "strings"
  "sync"

  "github.com/elastic/go-elasticsearch/v8"
  "github.com/elastic/go-elasticsearch/v8/esapi"
)

func main() {
  log.SetFlags(0)

  var (
    r  map[string]interface{}
    wg sync.WaitGroup
  )

  // Initialize a client with the default settings.
  //
  // An `ELASTICSEARCH_URL` environment variable will be used when exported.
  //
  es, err := elasticsearch.NewDefaultClient()
  if err != nil {
    log.Fatalf("Error creating the client: %s", err)
  }

  // 1. Get cluster info
  //
  res, err := es.Info()
  if err != nil {
    log.Fatalf("Error getting response: %s", err)
  }
  defer res.Body.Close()
  // Check response status
  if res.IsError() {
    log.Fatalf("Error: %s", res.String())
  }
  // Deserialize the response into a map.
  if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
    log.Fatalf("Error parsing the response body: %s", err)
  }
  // Print client and server version numbers.
  log.Printf("Client: %s", elasticsearch.Version)
  log.Printf("Server: %s", r["version"].(map[string]interface{})["number"])
  log.Println(strings.Repeat("~", 37))

  // 2. Index documents concurrently
  //
  for i, title := range []string{"Test One", "Test Two"} {
    wg.Add(1)

    go func(i int, title string) {
      defer wg.Done()

      // Build the request body.
      var b strings.Builder
      b.WriteString(`{"title" : "`)
      b.WriteString(title)
      b.WriteString(`"}`)

      // Set up the request object.
      req := esapi.IndexRequest{
        Index:      "test",
        DocumentID: strconv.Itoa(i + 1),
        Body:       strings.NewReader(b.String()),
        Refresh:    "true",
      }

      // Perform the request with the client.
      res, err := req.Do(context.Background(), es)
      if err != nil {
        log.Fatalf("Error getting response: %s", err)
      }
      defer res.Body.Close()

      if res.IsError() {
        log.Printf("[%s] Error indexing document ID=%d", res.Status(), i+1)
      } else {
        // Deserialize the response into a map.
        var r map[string]interface{}
        if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
          log.Printf("Error parsing the response body: %s", err)
        } else {
          // Print the response status and indexed document version.
          log.Printf("[%s] %s; version=%d", res.Status(), r["result"], int(r["_version"].(float64)))
        }
      }
    }(i, title)
  }
  wg.Wait()

  log.Println(strings.Repeat("-", 37))

  // 3. Search for the indexed documents
  //
  // Build the request body.
  var buf bytes.Buffer
  query := map[string]interface{}{
    "query": map[string]interface{}{
      "match": map[string]interface{}{
        "title": "test",
      },
    },
  }
  if err := json.NewEncoder(&buf).Encode(query); err != nil {
    log.Fatalf("Error encoding query: %s", err)
  }

  // Perform the search request.
  res, err = es.Search(
    es.Search.WithContext(context.Background()),
    es.Search.WithIndex("test"),
    es.Search.WithBody(&buf),
    es.Search.WithTrackTotalHits(true),
    es.Search.WithPretty(),
  )
  if err != nil {
    log.Fatalf("Error getting response: %s", err)
  }
  defer res.Body.Close()

  if res.IsError() {
    var e map[string]interface{}
    if err := json.NewDecoder(res.Body).Decode(&e); err != nil {
      log.Fatalf("Error parsing the response body: %s", err)
    } else {
      // Print the response status and error information.
      log.Fatalf("[%s] %s: %s",
        res.Status(),
        e["error"].(map[string]interface{})["type"],
        e["error"].(map[string]interface{})["reason"],
      )
    }
  }

  if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
    log.Fatalf("Error parsing the response body: %s", err)
  }
  // Print the response status, number of results, and request duration.
  log.Printf(
    "[%s] %d hits; took: %dms",
    res.Status(),
    int(r["hits"].(map[string]interface{})["total"].(map[string]interface{})["value"].(float64)),
    int(r["took"].(float64)),
  )
  // Print the ID and document source for each hit.
  for _, hit := range r["hits"].(map[string]interface{})["hits"].([]interface{}) {
    log.Printf(" * ID=%s, %s", hit.(map[string]interface{})["_id"], hit.(map[string]interface{})["_source"])
  }

  log.Println(strings.Repeat("=", 37))
}

// Client: 8.0.0-SNAPSHOT
// Server: 8.0.0-SNAPSHOT
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// [201 Created] updated; version=1
// [201 Created] updated; version=1
// -------------------------------------
// [200 OK] 2 hits; took: 5ms
//  * ID=1, map[title:Test One]
//  * ID=2, map[title:Test Two]
// =====================================

As you see in the example above, the esapi package allows to call the Elasticsearch APIs in two distinct ways: either by creating a struct, such as IndexRequest, and calling its Do() method by passing it a context and the client, or by calling the Search() function on the client directly, using the option functions such as WithIndex(). See more information and examples in the package documentation.

The elastictransport package handles the transfer of data to and from Elasticsearch, including retrying failed requests, keeping a connection pool, discovering cluster nodes and logging.

Read more about the client internals and usage in the following blog posts:

• https://www.elastic.co/blog/the-go-client-for-elasticsearch-introduction
• https://www.elastic.co/blog/the-go-client-for-elasticsearch-configuration-and-customization
• https://www.elastic.co/blog/the-go-client-for-elasticsearch-working-with-data

Helpers

The esutil package provides convenience helpers for working with the client. At the moment, it provides the esutil.JSONReader() and the esutil.BulkIndexer helpers.

Examples

The _examples folder contains a number of recipes and comprehensive examples to get you started with the client, including configuration and customization of the client, using a custom certificate authority (CA) for security (TLS), mocking the transport for unit tests, embedding the client in a custom type, building queries, performing requests individually and in bulk, and parsing the responses.

Author: Elastic
Source Code: https://github.com/elastic/go-elasticsearch 
License: Apache-2.0 License

#go #golang #client #elasticsearch 

Goes: A Library to interact with Elasticsearch in Go!

Goes : a library to interact with ElasticSearch

Supported operations

• index creation
• index removal
• simple indexing (document)
• bulk indexing
• search
• get

Example

You will find examples in example_test.go

Note: If you are switching to OwnLocal/goes from belogik/goes you will want to point whatever dependency management system you are using at the v1.0.0 tag to maintain backward compatibility (or change your code to support any API changes which may exist). The master branch will contain whatever the latest released version is and the develop branch will contain the currently in-progress version. Either of these branches may contain backwards-incompatible changes from the v1.0.0 tag (which contains the code as it was when the code was forked from belogik/goes).

Author: OwnLocal
Source Code: https://github.com/OwnLocal/goes 
License: BSD-3-Clause License

#go #golang 

Elasticsearch 7.x Backup — “Snapshot & Restore” on AWS S3

In 2016 I wrote an Article about Elasticsearch Backup, it had and still has quite good interests from people. I decided to start a new series of articles with the Backup topic as the main argument.

The old article covered Snapshot & Restore functionalities based on Elasticsearch 2.4.x and the upcoming version, the 5.0. As it was 4 years ago I choose to refresh this tutorial and making it the first of a series of more.

I will prepare a small article on how to use the snapshot & restore functionality with different cloud-provider. This article is based on Elasticsearch 7.x, it doesn’t mean it couldn’t work on older versions but I focused on the latest one.

Elasticsearch Snapshot & Restore

Elasticsearch has a smart solution to backup single indices or entire clusters to remote shared filesystem or S3 or HDFS. The snapshot ES creates does not so resource consuming and is relatively small.

The idea behind these snapshots is that they are not “archive” in a strict sense, these snapshots can only be read by a version of Elasticsearch that is capable to read the index version stored inside the snapshot.

So you can follow this quick scheme if you want to restore ES snapshots :

• A snapshot of an index created in 6.x can be restored to 7.x.
• A snapshot of an index created in 5.x can be restored to 6.x.
• A snapshot of an index created in 2.x can be restored to 5.x.
• A snapshot of an index created in 1.x can be restored to 2.x.

Snapshots of indices created with ES 1.x cannot be restored to 5.x or 6.x, snapshots of indices created in 2.x cannot be restored to 6.x or 7.x, and snapshots of indices created in 5.x cannot be restored to 7.x or 8.x.

#elasticsearch-snapshot #elasticsearch-plugins #elasticsearch #backup #elasticsearch-backup #aws