Author Archives: paulborile

About paulborile

I’m an IT Professional with a good experience in management of IT companies and lot of experience in software development. I have been partner of a couple of startups, managed teams of up to 25 people, worked as a freelance and currently leading the Italian development team for ScientiaMobile Inc, a Reston (US) based startup focused on image optimizing CDN and mobile detection technologies and services. Born in Dearborn Michigan and living in Italy since many years now I speak fluently both English and Italian, studied French and learned some Russian while working for some time for a Olivetti/Aeroflot project. I play bass guitar and ride gravel bikes in my free time.

Golang ElasticSearch for beginners

(You may not be familiar with what elasticsearch is : for a good introduction checkout this post from Igor Kopanev)
Extracting data from ElasticSearch in golang is not a simple task. First of all you might not be the one who wrote data into it so these are some commands to find out how is data structured in your ES server :

List all your indexes

$ curl https://<your-server>/_aliases?pretty=true
{  
  "billingdata" : {
    "aliases" : { }
  },
  "checkpoints" : {
    "aliases" : { }
  }
}

Your indexes data structures can be queried with :

curl https://<your-server>/<index_name> 


Now that I had an idea of how data was structured I started looking at the ES official library ( https://github.com/elastic/go-elasticsearch/ ) to find out that it is probably not used that much directly. This a code example to execute a query :

  // Build the request body.
  var buf bytes.Buffer
  query := map[string]interface{}{
    "query": map[string]interface{}{
      "match": map[string]interface{}{
        "title": "test",
      },
    },
  }
  if err := json.NewEncoder(&buf).Encode(query); err != nil {
    log.Fatalf("Error encoding query: %s", err)
  }
  // Perform the search request.
  res, err = es.Search(
    es.Search.WithContext(context.Background()),
    es.Search.WithIndex("billingdata"),
    es.Search.WithBody(&buf),
    es.Search.WithTrackTotalHits(true),
    es.Search.WithPretty(),
  )

Then (God bless Oliver Eilhard) I ray of light came in through my window and I found this package : https://godoc.org/github.com/olivere/elastic that makes everything much more simple :

	searchResult, err := es6.Search().
		Index("billingdata").
		// Query(timerangeQuery). // if you need to filter
		Pretty(true).
		Do(context.Background()) // execute

Much better. If you need a time range query :

// sdate,edate are time.Time	
timerangeQuery := elastic.NewBoolQuery().
		Filter(elastic.NewRangeQuery("@timestamp").
		From(sdate).
		To(edate))

Or maybe you need to query by time range and field value :

companyQuery = elastic.NewTermQuery("company_id", CompanyId)
// combine queries :
CombinedQuery = elastic.NewBoolQuery()
CombinedQuery = CombinedQuery.Must(timerangeQuery).Must(companyQuery)

// now search using Query(CombinedQuery)

Setting up your ES Client is pretty easy :

	// Create a custom HTTP client to setup timeouts and TLS config
	// esHTTPClient := &http.Client{
	// 	Timeout: time.Second * 10,
	// 	Transport: &http.Transport{
	// 		Dial: (&net.Dialer{
	// 			Timeout: 5 * time.Second,
	// 		}).Dial,
	// 		TLSHandshakeTimeout: 5 * time.Second,
	// 		TLSClientConfig: &tls.Config{
	// 			InsecureSkipVerify: true,
	// 		},
	// 	},
	// }

	// Create a client
	client, err := elastic.NewClient(
		// elastic.SetHttpClient(esHTTPClient),
		elastic.SetSniff(false),
		elastic.SetURL("http://localhost:9200"))

You can also aggregate data easily, sum, avergage. I’ll come up with more examples as soon as I have some working code.

October reading list

“simplicity, or the art of maximizing the work not done” : I really like this sentence, from Marco Cecconi https://sklivvz.com/posts/consider-using-simple-models-instead. Kind of summarizes my thoughts around simplicity in software, removing accidental complexity, over engineering code, get rid of complex and slow learning curve tools and languages. Thanks to Orfware for reference to Marco.

CORS : https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

Golang elasticsearch API : kudos to Oliver Eilhard for his package! https://godoc.org/github.com/olivere/elastic

What is Software Development ?

According to Steve McConnell “Code Complete” there are plenty of metaphors out there:

This last view seems to be shared by many ( https://signalvnoise.com/posts/591-brainstorm-the-software-garden , Jeff Atwood likes it too https://blog.codinghorror.com/tending-your-software-garden/ and I feel close to him on really many things) and I tend to see it more close to what software development is today : a continuously changing (growing) artifact, not totally manageable, that requires constant maintenance (here the building metaphor fails a bit), designed and redesigned over time like a garden, subject to seasons (“lots of new requirements” season, “robustness and reliability” season).

September reading list

Nice article on gRPC in a js client, Envoy and go gRPC server https://medium.com/swlh/building-a-realtime-dashboard-with-reactjs-go-grpc-and-envoy-7be155dfabfb

https://blog.codinghorror.com/tending-your-software-garden/

https://blog.codinghorror.com/bridges-software-engineering-and-god/

https://signalvnoise.com/posts/591-brainstorm-the-software-garden

https://blog.codinghorror.com/the-big-ball-of-mud-and-other-architectural-disasters/

The reverse technology relapse

We all know how much the apollo computer technology impacted on consumer technology : https://www.computerweekly.com/news/252466699/How-Apollo-11-influenced-modern-computer-software-and-hardware

Before Apollo computers were huge and filled whole rooms like this :

Burroughs B3500


Apollo guidance system was this :

Space industry led the way to a complete new era of digital personal computers.

We are now in 2020 and we all saw the first launch of a manned private mission to Space Station : the SpaceX dragon capsule attacched to ISS and astronauts @AstroBehnken and @Astro_Doug used a docking interface on a touchscreen device (try it out here https://iss-sim.spacex.com ) to dock dragon.

So consumer/private technology is currently used in the space industry : the complete opposite of Apollo/GoToTheMoon era.

Think of this from a political point of view : there were times when the biggest achievements in technology were driven by US federal agencies, pouring public money into research, universities, private companies to reach unbelievable goals (go to the moon). Those investment has been around $280 billion of today money. This drove the country into years of technological leadership, leadership US is loosing against another country (china) which is doing exactly what US did during the 60s i.e. pouring money into the economy with state participated companies.

Its time for another “go to the moon” type of effort and the new moon is giving this planet sustainable energy for sustainable growth.

Reaction/rejection to complexity and “worse is better”

Some points to further investigate : the main idea is that early 2000 years started a reaction to (unnecessary) complexity :

start here : https://medium.com/better-programming/how-unix-works-everything-you-were-too-afraid-to-ask-f8396aeb2763

simple is better (worse is better) : https://blog.codinghorror.com/worse-is-better/ the point here is all about what is considered to be better. Some furher reading “what’s wrong with word simple

Why use inheritance when a simple object will do : https://www.philweber.com/2002/11/is_inheritance_overrated

optimism in coding is dangerous : https://blog.codinghorror.com/defeating-optimism/

worse is better : https://www.dreamsongs.com/Files/WorseIsBetterPositionPaper.pdf

worse is better http://wiki.c2.com/?WorseIsBetter

again : https://www.jwz.org/doc/worse-is-better.html

Some covid-19 data for Italy

As you probably know Italy is experiencing one of the worst periods of its recent history. The covid-19 epidemic has put the medical system on its knees and the lock down is severely testing the population’s self-control capabilities.

Open data for the covid-19 infection is available here https://github.com/pcm-dpc/COVID-19 and updated daily. These are numbers and as usual they need to be read, understodd. One of the questions I was asking my self is how the rate of positives compared to the number of tests done is moving. I made this chart :

The ordinate is the percentage of positive cases against the number of tests done each day. Spikes of 42% are probably due to differences of response times for the laboratories which currently do not guarantee a stable processing time for tests. This behavior could be that due to the test methodology being changed or simply because we are finding less people infected.

Another information that can be interesting is the number of daily people being treated with intensive care :

Again here we might be in a situation of saturation of Intensive care units or just the number of people being treated in intensive care is decreasing.

This is the behavior of the number of people being hospitalized

Mar 2020 reading list

Image optimization and compression is one of main features of ImageEngine CDN (I work on this project). This Netflix blog article is great at going into details of why we probably need a new, modern image format : “AVIF for Next-Generation Image Coding” https://netflixtechblog.com/avif-for-next-generation-image-coding-b1d75675fe4

Faster Paint Metrics with Responsive Image Optimization CDNs
https://medium.com/@firt/faster-paint-metrics-with-responsive-image-optimization-cdns-d43340d4a48c

Speed up Woocommerce with an Image CDN https://hackernoon.com/speed-up-woocommerce-with-an-image-cdn-kt1q3yeg

Faster Image delivery with Chrome Light mode https://medium.com/imageengine/faster-image-delivery-with-chromes-lite-mode-93900eac126e

How to Reduce the Carbon Footprint of Your Website by 50% https://imageengine.io/blog/how-to-reduce-the-carbon-footprint-of-your-website-by-50/

Images are not static content https://css-tricks.com/images-are-not-static-content/

Prepare for the Ultimate Gaslighting https://forge.medium.com/prepare-for-the-ultimate-gaslighting-6a8ce3f0a0e0

Feb 2020 reading list

Python package formats : it’s hard to keep track of them 🙂

https://www.bernat.tech/pep-517-and-python-packaging/

Always though that variables shadowing was a problem : https://doc.rust-lang.org/book/ch03-01-variables-and-mutability.html#shadowing
and I still thinks so : https://mail.mozilla.org/pipermail/rust-dev/2013-May/004306.html

Microservice design : brokerless vs. broker/message bus based, interesting read

Nice dependency injection in Go : https://medium.com/@fsufitch/dependency-injection-and-testability-in-a-go-webservice-a91d0e5469dd

API Gateways : https://konghq.com/ https://tyk.io/

Goroutines in go 1.13 how they work : https://medium.com/a-journey-with-go/go-what-does-a-goroutine-switch-actually-involve-394c202dddb7

Monitoring air quality : http://www.esa.int/Applications/Observing_the_Earth/Copernicus/Monitoring_air_quality_takes_next_step