Golang ElasticSearch for beginners

(You may not be familiar with what elasticsearch is : for a good introduction checkout this post from Igor Kopanev)
Extracting data from ElasticSearch in golang is not a simple task. First of all you might not be the one who wrote data into it so these are some commands to find out how is data structured in your ES server :

List all your indexes

$ curl https://<your-server>/_aliases?pretty=true
{  
  "billingdata" : {
    "aliases" : { }
  },
  "checkpoints" : {
    "aliases" : { }
  }
}

Your indexes data structures can be queried with :

curl https://<your-server>/<index_name> 


Now that I had an idea of how data was structured I started looking at the ES official library ( https://github.com/elastic/go-elasticsearch/ ) to find out that it is probably not used that much directly. This a code example to execute a query :

  // Build the request body.
  var buf bytes.Buffer
  query := map[string]interface{}{
    "query": map[string]interface{}{
      "match": map[string]interface{}{
        "title": "test",
      },
    },
  }
  if err := json.NewEncoder(&buf).Encode(query); err != nil {
    log.Fatalf("Error encoding query: %s", err)
  }
  // Perform the search request.
  res, err = es.Search(
    es.Search.WithContext(context.Background()),
    es.Search.WithIndex("billingdata"),
    es.Search.WithBody(&buf),
    es.Search.WithTrackTotalHits(true),
    es.Search.WithPretty(),
  )

Then (God bless Oliver Eilhard) I ray of light came in through my window and I found this package : https://godoc.org/github.com/olivere/elastic that makes everything much more simple :

	searchResult, err := es6.Search().
		Index("billingdata").
		// Query(timerangeQuery). // if you need to filter
		Pretty(true).
		Do(context.Background()) // execute

Much better. If you need a time range query :

// sdate,edate are time.Time	
timerangeQuery := elastic.NewBoolQuery().
		Filter(elastic.NewRangeQuery("@timestamp").
		From(sdate).
		To(edate))

Or maybe you need to query by time range and field value :

companyQuery = elastic.NewTermQuery("company_id", CompanyId)
// combine queries :
CombinedQuery = elastic.NewBoolQuery()
CombinedQuery = CombinedQuery.Must(timerangeQuery).Must(companyQuery)

// now search using Query(CombinedQuery)

Setting up your ES Client is pretty easy :

	// Create a custom HTTP client to setup timeouts and TLS config
	// esHTTPClient := &http.Client{
	// 	Timeout: time.Second * 10,
	// 	Transport: &http.Transport{
	// 		Dial: (&net.Dialer{
	// 			Timeout: 5 * time.Second,
	// 		}).Dial,
	// 		TLSHandshakeTimeout: 5 * time.Second,
	// 		TLSClientConfig: &tls.Config{
	// 			InsecureSkipVerify: true,
	// 		},
	// 	},
	// }

	// Create a client
	client, err := elastic.NewClient(
		// elastic.SetHttpClient(esHTTPClient),
		elastic.SetSniff(false),
		elastic.SetURL("http://localhost:9200"))

You can also aggregate data easily, sum, avergage. I’ll come up with more examples as soon as I have some working code.

October reading list

“simplicity, or the art of maximizing the work not done” : I really like this sentence, from Marco Cecconi https://sklivvz.com/posts/consider-using-simple-models-instead. Kind of summarizes my thoughts around simplicity in software, removing accidental complexity, over engineering code, get rid of complex and slow learning curve tools and languages. Thanks to Orfware for reference to Marco.

CORS : https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

Golang elasticsearch API : kudos to Oliver Eilhard for his package! https://godoc.org/github.com/olivere/elastic

What is Software Development ?

According to Steve McConnell “Code Complete” there are plenty of metaphors out there:

This last view seems to be shared by many ( https://signalvnoise.com/posts/591-brainstorm-the-software-garden , Jeff Atwood likes it too https://blog.codinghorror.com/tending-your-software-garden/ and I feel close to him on really many things) and I tend to see it more close to what software development is today : a continuously changing (growing) artifact, not totally manageable, that requires constant maintenance (here the building metaphor fails a bit), designed and redesigned over time like a garden, subject to seasons (“lots of new requirements” season, “robustness and reliability” season).

September reading list

Nice article on gRPC in a js client, Envoy and go gRPC server https://medium.com/swlh/building-a-realtime-dashboard-with-reactjs-go-grpc-and-envoy-7be155dfabfb

https://blog.codinghorror.com/tending-your-software-garden/

https://blog.codinghorror.com/bridges-software-engineering-and-god/

https://signalvnoise.com/posts/591-brainstorm-the-software-garden

https://blog.codinghorror.com/the-big-ball-of-mud-and-other-architectural-disasters/