6 annoyances in GoLang

I still love it though

Go has been around for quite a while now and has had a lot of time to grow and mature, and it is really quite a lovely language. Watching it develop since its inception has been a delight. Its native concurrency is best in the industry, it is designed for things to be super-clean and standardized, and it compiles really fast. The support for it in IntelliJ/GoLand is top-notch and makes it really a blast to work in.

There are of course a lot of design decisions in the language that could be debated, like how setting a variable has to be a top level statement that cannot be embedded in other statements, how there is no “real” class based system, and how their specific name casing conventions are actually built into the language and enforced. However, I understand the reason for all of these decisions and feel that the designers of the language were justified in the decisions they made for the reasons they give.

There are at least 6 problems I’ve run into a lot recently though that I feel could be better. I would love to jump into the source of the language and see about making these fixes myself, but the GoLang team has a history of completely ignoring outsiders, and it would be highly unlikely that they would accept anything I submitted, especially since a number of these things would require very long and complicated proposals to even start looking at. Go may be an open source language, but it is not open development.

These first 3 would be additions to the language that would not break anything, which is in line with their promise of never breaking anything.

1) Interface with constraints elements cannot be used as variable types. Example:

type intish interface{ int64 | uint64 }
func adder[T intish](val T) T { return val + 1 } //This is allowed
var val intish //Error: cannot use type intish outside a type constraint: interface contains type constraints

IntelliJ inspection Error: Interface includes constraint elements '...', can only be used in type parameters

It wouldn’t be a big change for the compiler and runtime to be able to enforce these types of constraints when typecasting and I really wish the language had this. This is the type of thing TypeScript does beautifully since it is actually part of its native design and functioning.

2) Member functions (methods with receivers) cannot have generics

type foobar int
func (*foo) bar[T any](val T) {} //Method cannot have type parameters

3) []generic cannot be typecast to []any

There is no real reason that an array containing interfaces shouldn’t be convertable via typecast to another array of interface types as long as they are compatible.

type testAny any
startVar := make([]testAny, 3)
endVar := []any(startVar) //cannot convert startVar (variable of type []testAny) to type []any

---

This next one is an unfortunate consequence of the language design and would not be easily fixable.

4) Import cycle not allowed

This means there cannot be circular dependencies. If package A includes package B, then package B cannot include package A. This often wasn’t a problem with languages like C since the headers were independent of the implementations, but because packages in Go do not have headers, and their package types cannot be used as method receivers in other packages, this one is just not possible. And it can be quite limiting.

---

This next one is a bug in either the language or the documentation, and I consider it to be a security problem, but the go team does not seem to care about it. See https://github.com/golang/go/issues/65201

[Edit on 2024-04-10] Oh wow! they fixed it!

5) sql.RawBytes modifies the current buffers instead of setting new buffers like the documentation says.

All the documentation says about RawBytes is “RawBytes is a byte slice that holds a reference to memory owned by the database itself. After a Rows.Scan into a RawBytes, the slice is only valid until the next call to Rows.Next, Rows.Scan, or Rows.Close.”

If the value you are reading is a []byte/string then reading into a RawBytes works as expected. However, if it is anything else, like an int, it reads into the buffer that RawBytes already holds. This can lead to buffer injections and other really nasty bugs. For example:

db.Exec(`CREATE TEMPORARY TABLE goTest (i int NOT NULL, str varchar(10)) ENGINE=MEMORY`)
db.Exec(`INSERT INTO goTest VALUES (?, ?)`, 6, "foobar")
var scanIn sql.RawBytes
rows, _ := db.Query(`SELECT str FROM goTest WHERE i=?`, 6)
rows.Next()
rows.Scan(&scanIn)
//Do something with scanIn
rows.Close()
rows, _ = db.Query(`SELECT i FROM goTest WHERE i=?`, 6)
rows.Next()
rows.Scan(&scanIn) //This corrupts the internal sql driver buffer since it reads an int into the pointer we received earlier

---

This final one has been an annoyance of mine since day 1 of working with Go, and it still annoys the hell out of me. There could be ways to fix it without breaking things, but I cannot think of any truly elegant solutions.

6) Short variable declaration can’t handle setting both new and already existing variables
This is the most common pattern you’ll see in Go by far:

var data string
if _data, err := someFunc(); err != nil {
    fmt.Println(err)
} else {
    data = _data
}

You can either do it this way, or also declare err in the outer scope, thereby polluting the scope with unneeded variables. There is no way to set both the temporary error variable and also set the outer data variable in 1 line.

One non-breaking hack would be to add a symbol before variables you wanted to set in the outer scope. Example:

var data string
if +data, err := someFunc(); err != nil {
    fmt.Println(err)
}

Results from my first high-load scalable system

Putting the cloud on a scale so it’s not so heavy

I’ve wanted to create and test a large-scale application for a very long time but have never really had the chance until recently. The Vintage Experience project I did earlier this year finally gave me the opportunity. As one of many parts of the project, I was tasked to create a voting system that could handle 1 million votes via a web page in a 30 second time span. The final system was deployed successfully without any problems for Gala Artis 2013 (a French Canadian artist/TV awards show). The following are the results of my implementation and testing.

The main front-end was done via a static HTML page (smart-phone optimized) that was hosted by Amazon S3, where handling 33k requests/second is a drop in the bucket. All voting requests were done via AJAX from this web page to backend servers hosted by Amazon EC2.

The backend was programmed in GoLang as a simple web server, optimized for memory and speed, which spawned a new goroutine for each incoming request. The request returned a message to the user stating either the error message, or success if the vote was added to the database. Each server held a single persistent MySQL connection to an Amazon RDS “Large DB Instance” with the minimum IOPS (1000). Votes from a server were sent to the database in batches once a second, or earlier if 10,000 votes had been received.

The servers were Amazon “M1 Standard Extra Large” (m1.xlarge) instances running Linux, of which there were 6 total, handling vote requests delegated by a round-robin DNS on Amazon’s Route 53. During stress testing, each server was found to be able to handle 6800 requests/second, and load was staying under 3, so there were was probably another bottle neck. Running the same tests using php(sapi)+apache(fork), only 4500 requests/second could be executed, and there was a 16+ load.

On the servers, I found it necessary to set the following sysctl setting to increase performance “net.core.somaxconn=1024”. The following commands need to be run to execute this:

sysctl 'net.core.somaxconn=1024' #Store for the current computer session
echo 'net.core.somaxconn=1024' >> /etc/sysctl.conf #Set after a reboot

Stress test client instances were also run on Amazon as m1.xlarge instances, and were found to be able to push 5000-6000 requests/second. The GoLang test clients spawned 200 requests at a time and waited for them to finish before sending the next batch. The client system needed the following sysctl settings for optimal performance:

net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.ip_local_port_range="15000 65534"

Cygwin SIGINT fix for golang

Cygwin has had a long time problem that, depending on your configuration, may cause you to be unable to send a SIGINT (interrupt signal via Ctrl+C) to a native Windows command line executables. As a matter of fact, trying to do so may completely freeze up the console, requiring a process kill of the actual console, bash, and the executable you ran. This problem can crop up for many reasons including the version of Cygwin you are running and your terminal emulator. I specifically installed mintty as my default Cygwin console to get rid of this problem a long time ago (among many other features it had), and now it even has this problem.

While my normal solution is to try and steer clear of native Windows command line executables in Cygwin, this is not always an option. Golang was also causing me this problem every time I ran a network server, which was especially problematic as I would have to ALSO manually kill the server process or it would continue to hold the network port so another test of the code could not use it. An example piece of code is as follows:

package main
import ( "net/http"; "fmt" )
func main() {
	var HR HandleRequest
	if err := http.ListenAndServe("127.0.0.1:81", HR); err!=nil {
		fmt.Println("Error starting server") }
}

//Handle a server request
type HandleRequest struct{}
func (HR HandleRequest) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	fmt.Printf("Received connection from: %s\n", req.RemoteAddr)
}

---

go run example.go

The first solution I found to this problem, which is by far the best solution, was to build the executable and then run it, instead of just running it straight from go.

go build example.go && example.exe

However, as of this post, it seems to no longer work! The last time I tested it and confirmed it was working was about 3 months ago, so who knows what has changed since then.

The second solution is to just build in some method of killing the process that uses “os.Exit”. For example, the following will exit if the user types “exit”

func ListenForExitCommand() {
	for s:=""; s!="exit"; { //Listen for the user to type exit
		if _, err:=fmt.Scanln(&s); err!=nil {
			if err.Error()!="unexpected newline" {
				fmt.Println(err) }
		} else if s=="flush" || s=="exit" {
			//Clean up everything here
		}
	}
	fmt.Println("Exit received, closing process")
	os.Exit(1)
}

and then add the following at the top of the main function:

go ListenForExitCommand() //Listen for "exit" command

Optionally encrypted TCP class for Google's Go

Yet another new language to play with

I wanted to play around with Google's go language a little so I ended up decided on making a simple class that helps create a TCP connection between a server and client that is encrypted via TLS, or not, depending upon a flag. Having the ability to not encrypt a connection is useful for debugging and testing purposes, especially if other people are needing to create clients to connect to your server.

The example server.go file listens on port 16001 and for every set of data it receives, it sends the reversed string back to the client. (Note there are limitations to the string lengths in the examples due to buffer and packet payload length restrictions).

The example client.go file connects to the server (given via the 1st command line parameter), optionally encrypts the connection (depending upon the 2nd command line parameter), and sends the rest of the parameters to the server as strings.

The encryptedtcp.go class has the following exported functions:

• StartServer: Goes into a connection accepting loop. Whenever a connection is accepted, it checks the data stream for either the "ENCR" or "PTXT" flags, which control whether a TLS connection is created or not. The passed "clientHandler" function is called once the connection is completed.
• StartClient: Connects to a server, passes either the "ENCR" or "PTXT" flag as noted above, and returns the finished connection.

Connections are returned as "ReadWriteClose" interfaces. Creating the pem and key certificate files is done via openssl. You can just google for examples.

server.go:

package main
import ( "./encryptedtcp"; "fmt"; "log" )

func main() {
	if err := encryptedtcp.StartServer("server.pem", "server.key", "0.0.0.0:16001", handleClient); err != nil {
		log.Printf("%q\n", err) }
}

func handleClient(conn encryptedtcp.ReadWriteClose) {
	buf := make([]byte, 512)
	for {
		//Read data
		n, err := conn.Read(buf)
		if err != nil {
			log.Printf("Error Reading: %q\n", err); break }
		fmt.Printf("Received: %q\n", string(buf[:n]))

		//Reverse data
		for i, m := 0, n/2; i<m; i++ { //Iterate over half the list
			buf[i], buf[n-i-1] = buf[n-i-1], buf[i] } //Swap first and half of list 1 char at a time

		//Echo back reversed data
		n, err = conn.Write(buf[:n])
		if err != nil {
			log.Printf("Error Writing: %q\n", err); break }
		fmt.Printf("Sent: %q\n", string(buf[:n]))
	}
}

client.go:

package main
import ( "./encryptedtcp"; "fmt"; "log"; "os" )

func main() {
	//Confirm parameters, and if invalid, print the help
	if len(os.Args) < 4 || (os.Args[2] != "y" && os.Args[2] != "n") {
		log.Print("First Parameter: ip address to connect to\nSecond Parameter: y = encrypted, n = unencrypted\nAdditional Parameters (at least 1 required): messages to send\n"); return }

	//Initialize the connection
	conn, err := encryptedtcp.StartClient("client.pem", "client.key", os.Args[1]+":16001", os.Args[2]=="y" )
	if err != nil {
		log.Printf("%q\n", err); return }
	defer conn.Close()

	//Process all parameters past the first
	buf := make([]byte, 512)
	for _, msg := range os.Args[3:] {
		//Send the parameter
		if(len(msg)==0) {
			continue }
		n, err := conn.Write([]byte(msg))
		if err != nil {
			log.Printf("Error Writing: %q\n", err); break }
		fmt.Printf("Sent: %q\n", msg[:n])

		//Receive the reply
		n, err = conn.Read(buf)
		if err != nil {
			log.Printf("Error Reading: %q\n", err); break }
		fmt.Printf("Received: %q\n", string(buf[:n]))
	}
}

encryptedtcp/encryptedtcp.go:

//A simple TCP client/server that can be encrypted (via tls) or not, depending on a flag passed from the client

package encryptedtcp

import ( "crypto/rand"; "crypto/tls"; "net"; "log" )

//Goes into a loop to accept clients. Returns a string on error
func StartServer(certFile, keyFile, listenOn string, clientHandler func(ReadWriteClose)) (error) {
	//Configure the certificate information
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return MyError{"Cannot Load Keys", err} }
	conf := tls.Config{Certificates:[]tls.Certificate{cert}, ClientAuth:tls.RequireAnyClientCert, Rand:rand.Reader}

	//Create the listener
	listener, err := net.Listen("tcp", listenOn)
	if err != nil {
		return MyError{"Cannot Listen", err} }
	defer listener.Close()

	//Listen and dispatch clients
	for {
		conn, err := listener.Accept()
		if err != nil {
			return MyError{"Cannot Accept Client", err} }
		go startHandleClient(conn, &conf, clientHandler)
	}

	//No error to return - This state is unreachable in the current library
	return nil
}

//Return the io stream for the connected client
func startHandleClient(conn net.Conn, conf* tls.Config, clientHandler func(ReadWriteClose)) {
	defer conn.Close()

	//Confirm encrypted connection flag (ENCR = yes, PTXT = no)
	isEncrypted := make([]byte, 4)
	amountRead, err := conn.Read(isEncrypted)
	if err != nil {
		log.Printf("Cannot get Encrypted Flag: %q\n", err); return }
	if amountRead != 4 {
		log.Printf("Cannot get Encrypted Flag: %q\n", "Invalid flag length"); return }
	if string(isEncrypted) == "PTXT" { //If plain text, just pass the net.Conn object to the client handler
		clientHandler(conn); return
	} else if string(isEncrypted) != "ENCR" { //If not a valid flag value
		log.Printf("Invalid flag value: %q\n", isEncrypted); return }

	//Initialize the tls session
	tlsconn := tls.Server(conn, conf)
	defer tlsconn.Close()
	if err := tlsconn.Handshake(); err != nil {
		log.Printf("TLS handshake failed: %q\n", err); return }

	//Pass the tls.Conn object to the client handler
	clientHandler(tlsconn)
}

//Start a client connection
func StartClient(certFile, keyFile, connectTo string, isEncrypted bool) (ReadWriteClose, error) {
	//Configure the certificate information
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return nil, MyError{"Cannot Load Keys", err} }
	conf := tls.Config{Certificates:[]tls.Certificate{cert}, InsecureSkipVerify:true}

	//Connect to the server
	tcpconn, err := net.Dial("tcp", connectTo)
	if err != nil {
		return nil, MyError{"Cannot Connect", err} }

	//Handle unencrypted connections
	if !isEncrypted {
		tcpconn.Write([]byte("PTXT"))
		return tcpconn, nil //Return the base tcp connection
	}

	//Initialize encrypted connections
	tcpconn.Write([]byte("ENCR"))
	conn := tls.Client(tcpconn, &conf)
	conn.Handshake()

	//Confirm handshake was successful
	state := conn.ConnectionState()
	if !state.HandshakeComplete || !state.NegotiatedProtocolIsMutual {
		conn.Close()
		if !state.HandshakeComplete {
			return nil, MyError{"Handshake did not complete successfully", nil}
		} else {
			return nil, MyError{"Negotiated Protocol Is Not Mutual", nil} }
	}

	//Return the tls connection
	return conn, nil
}

//Error handling
type MyError struct {
	Context string
	TheError error
}
func (e MyError) Error() string {
	return e.Context+": "+e.TheError.Error(); }

//Interface for socket objects (read, write, close)
type ReadWriteClose interface {
	Read(b []byte) (n int, err error)
	Write(b []byte) (n int, err error)
	Close() error
}