2022 Reading list (updated as we go)

Some tools for enumerating : https://github.com/OJ/gobuster, https://github.com/OWASP/Amass

Updates on the twitter affair : https://newsletter.pragmaticengineer.com/p/the-scoop-twitters-ongoing-cruel

Reverse engineering Swift physics

Pelikan cache, a C/Rust memcached type cache server made in twitter http://www.pelikan.io/

Giza 3D http://giza.fas.harvard.edu/giza3d/

Memory allocator used in Go language : https://google.github.io/tcmalloc/design.html
Slab Allocation : https://hammertux.github.io/slab-allocator
Linux virtual FS filename lookup : https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html
Linux dentry cache and slabtop command

Your wordpress site can feed mastodon https://fedi.tips/the-fediverse-beyond-mastodon/#WordPress

Image Format wars :
– Google removing JXL support from chrome : https://chromium-review.googlesource.com/c/chromium/src/+/3988999
– reasons https://www.phoronix.com/news/Chrome-Dropping-JPEG-XL-Reasons
– comments : https://cloudinary.com/blog/the-case-for-jpeg-xl
– support for JXL in android : https://issuetracker.google.com/issues/259900694?pli=1

Bare metal or cloud services : https://levelup.gitconnected.com/how-we-reduced-our-annual-server-costs-by-80-from-1m-to-200k-by-moving-away-from-aws-2b98cbd21b46

New Directions in Cloud Programming : https://www.cidrdb.org/cidr2021/papers/cidr2021_paper16.pdf

I know you all know but, just a reminder : 

package main

import "fmt"

func main() {
  var m = 0.1
  var n = 0.2
  fmt.Println(m + n)

  var a = 0.1
  var b = 6.0
  fmt.Println(a * b)

  var o = 0.6
  var p = 0.3
  fmt.Println(o + p)

  var f = 1.39
  fmt.Println(f * f)

  const c = 1.39
  fmt.Println(c * c)
}

0.30000000000000004
0.6000000000000001
0.8999999999999999
1.9320999999999997
1.9321

Reducing Cognitive load (Golab2022 notes) in code :

  • Line of Sight : 1st level, second level for error conditions
    • eliminate else, return early, avoid extra nesting, wrap in function
    • align the happy path
    • remove misterious booleans
  • Package Names : concise and clear, move on the calling side. No util/common/ packages, split
  • Pure functions as much as possible
  • pass with & to indicate we change the object
  • Don’t lie to the reader

Interesting WASM talk from https://xeiaso.net/

Organize you software assets https://stackoverflow.blog/2022/09/19/i-spent-two-years-trying-to-do-what-backstage-does-for-free/ spotify open sourced their internal tool

VCF East 2019 Bryan Kernigham interviews Ken Thompson : how unix, pipes, and other things like grep came out just to run nroff/troff for Bell Labs patents office https://youtu.be/EY6q5dv_B-o

Engineering ladders https://github.com/jorgef/engineeringladders

Stackoverflow engineering https://hanselminutes.com/847/engineering-stack-overflow-with-roberta-arcoverde

Stackexchange 1.3 bil pageview/month : https://stackexchange.com/performance

Airbnb path from monolith to microservices https://medium.com/qe-unit/airbnbs-microservices-architecture-journey-to-quality-engineering-d5a490e6ba4f

Perche’ l’innovazione e’ cosi’ difficile in italia : qui

Questo non vale solo per la digitalizzazione, vale per l’industrializzazione e per il capitalismo. Prendo in prestito la domanda (e la risposta) di Roberto Maragliano “Perché i temi della maturità sono ancora così? Ce lo dice Camillo Olivetti (padre di Adriano) in un appunto del 1927: “L’istruzione della nostra borghesia ha un fondamento prettamente anti-industriale. Noi siamo ancora i figli dei Latini che lasciarono ai servi e ai liberti i lavori industriali e che in ben poco conto li ritennero, tanto che ci tramandarono i nomi dei più mediocri proconsoli e dei poetucoli ed istrioni che dilettarono la decadenza romana, ma non ci ricordarono neppure i nomi di quei sommi ingegneri che costruirono le strade gli acquedotti e i grandi monumenti dell’Impero Romano” (da Paolo Bricco, Adriano Olivetti, un italiano del Novecento, Rizzoli, appena uscito)”.

End to end tracing : Jaeger (opentelemetry)

Consistent Hashing used for load balancing resources :

Porting python2 code to python3

Interesting point of view on “complete rewrite” vs “enhance/evolve the existing” from dropbox tech blog

T Shape Engineers

Perkeep .. “Throughout our life, we all continue to generate content …”

Peer to peer hypermedia protocol : https://ipfs.io/

Building and Testing in-kernel (Rust code to be discussed at next Linux Plumbers : https://lpc.events/event/16/page/193-proposed-microconferences and discussed here : https://arstechnica.com/gadgets/2021/03/linus-torvalds-weighs-in-on-rust-language-in-the-linux-kernel/

Linux Foundation Real Time OS : https://www.zephyrproject.org/learn-about/

Microservice patterns : https://microservices.io/patterns/index.html

Broken Window Theory applied to software ( initial statement in “The Pragmatic Programmer” by Andrew Hunt and David Thomas, 1999 if I’m not wrong) interesting review and interesting to meet another fan of simplicity and minimalism in code.

Again on avoiding accidental complexity (I wrote something on this in the past) https://searchsoftwarequality.techtarget.com/tip/How-to-prevent-accidental-complexity-in-software-development
Interesting :

Super, 10X, rockstar — these are names for developers who produce a lot of code or work in many development areas. Such developers sometimes build a reputation — and an ego. “These super developers usually make their code unnecessarily complex just because they can,” Rials said. Other team members struggle to comprehend and edit the complex code. These colleagues might even compliment the super developer about the code’s complexity, which encourages more of it. Businesses should empower junior developers to take a fresh look at code.

Bill Rials, associate director and professor at Tulane University’s School of Professional Advancement

How common the pattern above is ..

Composer, music theorist, architect, performance director and engineer https://en.wikipedia.org/wiki/Iannis_Xenakis (I love multi potential people). Known for his studies on Music generation software : author of UPIC, which inspired many other tools like Iannix, HighC, UPISketch. I was able to run HighC …

Tz0, a video game made to listen music

The first ecologist ? https://www.rachelcarson.org/

Clearness when she explains, clearness of vision, clearness of thinking : Mariana Mazzucato

Back to untargeted adv ?

https://projecteuler.net/ take some time to solve math things

The importance of Rubber Ducking : a funny tool to do it alone ..

Ethical resources : sustainable, mindful, transparent, open and much more. Here’s a list of tools (browsers, search engines, email services, web hosting, tools for team collaboration, messaging, and a lot more) for a more ethical internet.


Enchanted by Amanda Gorman speech at Presidential Inauguration 2021

“When day comes we ask ourselves,

where can we find light in this never-ending shade?

The loss we carry,

a sea we must wade

We’ve braved the belly of the beast

We’ve learned that quiet isn’t always peace

And the norms and notions

of what just is

Isn’t always justice

And yet the dawn is ours

before we knew it

Somehow we do it

Somehow we’ve weathered and witnessed

a nation that isn’t broken

but simply unfinished

We the successors of a country and a time

Where a skinny Black girl

descended from slaves and raised by a single mother

can dream of becoming president

only to find herself reciting for one

And yes we are far from polished

far from pristine

but that doesn’t mean we are

striving to form a union that is perfect

We are striving to forge a union with purpose

To compose a country committed to all cultures, colors, characters and

conditions of man

And so we lift our gazes not to what stands between us

but what stands before us

We close the divide because we know, to put our future first,

we must first put our differences aside

We lay down our arms

so we can reach out our arms

to one another

We seek harm to none and harmony for all

Let the globe, if nothing else, say this is true:

That even as we grieved, we grew

That even as we hurt, we hoped

That even as we tired, we tried

That we’ll forever be tied together, victorious

Not because we will never again know defeat

but because we will never again sow division

Scripture tells us to envision

that everyone shall sit under their own vine and fig tree

And no one shall make them afraid

If we’re to live up to our own time

Then victory won’t lie in the blade

But in all the bridges we’ve made

That is the promise to glade

The hill we climb

If only we dare

It’s because being American is more than a pride we inherit,

it’s the past we step into

and how we repair it

We’ve seen a force that would shatter our nation

rather than share it

Would destroy our country if it meant delaying democracy

And this effort very nearly succeeded

But while democracy can be periodically delayed

it can never be permanently defeated

In this truth

in this faith we trust

For while we have our eyes on the future

history has its eyes on us

This is the era of just redemption

We feared at its inception

We did not feel prepared to be the heirs

of such a terrifying hour

but within it we found the power

to author a new chapter

To offer hope and laughter to ourselves

So while once we asked,

how could we possibly prevail over catastrophe?

Now we assert

How could catastrophe possibly prevail over us?

We will not march back to what was

but move to what shall be

A country that is bruised but whole,

benevolent but bold,

fierce and free

We will not be turned around

or interrupted by intimidation

because we know our inaction and inertia

will be the inheritance of the next generation

Our blunders become their burdens

But one thing is certain:

If we merge mercy with might,

and might with right,

then love becomes our legacy

and change our children’s birthright

So let us leave behind a country

better than the one we were left with

Every breath from my bronze-pounded chest,

we will raise this wounded world into a wondrous one

We will rise from the gold-limbed hills of the west,

we will rise from the windswept northeast

where our forefathers first realized revolution

We will rise from the lake-rimmed cities of the midwestern states,

we will rise from the sunbaked south

We will rebuild, reconcile and recover

and every known nook of our nation and

every corner called our country,

our people diverse and beautiful will emerge,

battered and beautiful

When day comes we step out of the shade,

aflame and unafraid

The new dawn blooms as we free it

For there is always light,

if only we’re brave enough to see it

If only we’re brave enough to be it”

Interview with Ken Thompson

(this comes from a DrDobbs’s post May 18, 2011 – the article disappeared from the site so I’m just saving it here)

The creator of UNIX discusses writing UNIX, the Go language, and collaborating with Dennis Ritchie

The Japan Prize, one of the highest honors awarded for outstanding contribution to science and technology, was awarded jointly this year to Ken Thompson and Dennis Ritchie for the creation of UNIX. The prize is normally given to the recipients at a lavish banquet in Tokyo attended by the emperor. However, due to the April earthquake and tsunami, the prizes this year were distributed at the honorees’ place of work. I was able to attend the ceremony for Ken Thompson, held at Google headquarters, where he currently works. After the ceremony, he consented to this exclusive interview.

DDJ: Congratulations on winning this prize.

KT: Thanks.

Developing UNIX

DDJ: You’ve received a lot of awards over the years for UNIX. At what point in UNIX’s development did it become clear it was going to be something much bigger than you’d anticipated?

KT: The actual magnitute, that no one could have guessed. I gather it’s still growing now. I thought it would be useful to essentially anybody like me because it was not built for someone else or some third party. That was a perjorative term then. It was written for Dennis and me and our group to do its work. And I think it would have been useful to anybody who did the kind of work that we did. And therefore, I always thought it was something really good that was going to take off.

Especially the language [C]. The language grew up with one of the rewritings of the system and, as such, it became perfect for writing systems. We would change it daily as we ran into trouble building UNIX out of the language and we’d modify it for our needs.

DDJ: A symbiosis of sorts…

KT: Yeah. It became the perfect language for what it was designed to do. I always thought the language and the system were widely applicable.

DDJ: In the presentation today, it mentioned that UNIX was open source. Was UNIX open source from the beginning?

KT: Well there was no such term as “open source” then.

DDJ: I was under the impression that UNIX really became open source with the Berkeley distribution.

KT: No, we charged $100, which was essentially the reproduction cost of the tape, and then send it out. And we distributed, oh, probably close to 100 copies to universities and others.

Go Language

DDJ: Skipping several decades of work, let’s speak about Go. I was just at the Google I/O Conference, where it was announced that Go will be supported on the Google App Engine. Does that presage a wider adoption of Go within Google, or is it still experimental?

KT: It’s expanding every day and not being forced down anybody’s throat. It’s hard to adopt it to a project inside of Google because of the learning curve. It’s brand new and there aren’t good manuals for it, except what’s on the Web. And then, of course, its label of being experimental, so people are a little afraid. In spite of that, it’s growing very fast inside of Google.

DDJ: In the presentation before the awarding of the Japan Prize today, you were quoted on the distinction between reasearch and development. [The former, Thompson stated, was directionless, whereas development had a specific goal in mind.] So in that context, is Go experimental?

KT: Yes. When the three of us [Thompson, Rob Pike, and Robert Griesemer] got started, it was pure research. The three of us got together and decided that we hated C++. [laughter]

DDJ: I think there’d be a lot of people who are with you on that.

KT: It’s too complex. And going back, if we’d thought of it, we’d have done an object-oriented version of C back in the old days.

DDJ: You’re saying you would have?

KT: Yes, but we were not evangelists of object orientation. [Returning to Go,] we started off with the idea that all three of us had to be talked into every feature in the language, so there was no extraneous garbage put into the language for any reason.

DDJ: It’s a lean language, indeed.

Collaboration with Dennis Ritchie

DDJ: Returning to UNIX, for a moment, when you and Dennis worked together, how did that collaboration operate? Were you working side by side?

KT: I did the first of two or three versions of UNIX all alone. And Dennis became an evangelist. Then there was a rewrite in a higher-level language that would come to be called C. He worked mostly on the language and on the I/O system, and I worked on all the rest of the operating system. That was for the PDP-11, which was serendipitous, because that was the computer that took over the academic community.

DDJ: Right.

KT: We collaborated every day. There was a lunch that we went to. And we’d talk over lunch. Then, at night, we each worked from our separate homes but we were in constant communication. In those days, we had mail and writ (pronounced ‘write’), and writ would pop up on your screen and say there was a message from so-and-so.

DDJ: So, IM essentially.

KT: Yes, IM. There was no doubt about that! And we discussed things from home with writ. We worked very well together and didn’t collaborate a lot except to decide who was going to do what. Then we’d run and very independently do separate things. Rarely did we ever work on the same thing.

DDJ: Was there any concept of looking at each other’s code or doing code reviews?

KT: [Shaking head] We were all pretty good coders.

DDJ: I suspect you probably were! [Laughter]

SCM

DDJ: Did you use any kind of source code management product when working together?

KT: No, those products really came later; after UNIX. We had something like it, which we called “the code motel” because you could check your code in but you couldn’t check it out! So, really, no we didn’t.

DDJ: I bet you use SCM today in your work on Go.

KT: Oh, yes, Google makes us do that!

Golang ElasticSearch for beginners

(You may not be familiar with what elasticsearch is : for a good introduction checkout this post from Igor Kopanev)
Extracting data from ElasticSearch in golang is not a simple task. First of all you might not be the one who wrote data into it so these are some commands to find out how is data structured in your ES server :

List all your indexes

$ curl https://<your-server>/_aliases?pretty=true
{  
  "billingdata" : {
    "aliases" : { }
  },
  "checkpoints" : {
    "aliases" : { }
  }
}

Your indexes data structures can be queried with :

curl https://<your-server>/<index_name>


Now that I had an idea of how data was structured I started looking at the ES official library ( https://github.com/elastic/go-elasticsearch/ ) to find out that it is probably not used that much directly. This a code example to execute a query :

  // Build the request body.
  var buf bytes.Buffer
  query := map[string]interface{}{
    "query": map[string]interface{}{
      "match": map[string]interface{}{
        "title": "test",
      },
    },
  }
  if err := json.NewEncoder(&buf).Encode(query); err != nil {
    log.Fatalf("Error encoding query: %s", err)
  }
  // Perform the search request.
  res, err = es.Search(
    es.Search.WithContext(context.Background()),
    es.Search.WithIndex("billingdata"),
    es.Search.WithBody(&buf),
    es.Search.WithTrackTotalHits(true),
    es.Search.WithPretty(),
  )

Then (God bless Oliver Eilhard) I ray of light came in through my window and I found this package : https://godoc.org/github.com/olivere/elastic that makes everything much more simple :

	searchResult, err := es6.Search().
		Index("billingdata").
		// Query(timerangeQuery). // if you need to filter
		Pretty(true).
		Do(context.Background()) // execute

Much better. If you need a time range query :

// sdate,edate are time.Time	
timerangeQuery := elastic.NewBoolQuery().
		Filter(elastic.NewRangeQuery("@timestamp").
		From(sdate).
		To(edate))

Or maybe you need to query by time range and field value :

companyQuery = elastic.NewTermQuery("company_id", CompanyId)
// combine queries :
CombinedQuery = elastic.NewBoolQuery()
CombinedQuery = CombinedQuery.Must(timerangeQuery).Must(companyQuery)

// now search using Query(CombinedQuery)

Setting up your ES Client is pretty easy : 

	// Create a custom HTTP client to setup timeouts and TLS config
	// esHTTPClient := &http.Client{
	// 	Timeout: time.Second * 10,
	// 	Transport: &http.Transport{
	// 		Dial: (&net.Dialer{
	// 			Timeout: 5 * time.Second,
	// 		}).Dial,
	// 		TLSHandshakeTimeout: 5 * time.Second,
	// 		TLSClientConfig: &tls.Config{
	// 			InsecureSkipVerify: true,
	// 		},
	// 	},
	// }

	// Create a client
	client, err := elastic.NewClient(
		// elastic.SetHttpClient(esHTTPClient),
		elastic.SetSniff(false),
		elastic.SetURL("http://localhost:9200"))

You can also aggregate data easily, sum, avergage. I’ll come up with more examples as soon as I have some working code.