[This is an extended version of a previous post I made directly to Google+.]
I wrote a program in Go that clusters unstructured data and involves a lot of multi-dimensional distance calculations. You can see the ongoing process at https://github.com/bobhancock/goxmeans.git. It contains a pipeline that prepares jobs, fans out to worker goroutines (one for each CPU) and stores the results. All communication is done with channels.
I’ll write more about goxmeans in the future.
I started some performance testing with relatively small datasets of 250,000 points and 2-6 centroids and it worked as expected. When I increased the number of centroids to 9, the program would run for a while, but then my laptop would shut down. My first thought was that there was some sort of bug with memory or the underlying mechanism of pointers.
I viewed syslog and it showed that the Intel ACPI controller registered that the temperature had hit 100 degrees celsius. psensor showed that it had hit a max of 94 degrees--still high compared to the average of 55 but not deadly. In order to save my hardware, it did not issue any warnings, it immediately shut down the computer.
I've never had this type of problem with C++ or Python. Go was so efficiently feeding jobs to the CPUs that the temperature rose almost 100% and stayed there. I reset the fan to manual and ran it at max and everything worked fine, however, to be safe I reset the default to auto. I'll perform the rest of the performance tests on a larger machine.
If I had tried to write this in C++ with threads it would have taken much, much longer and experience makes me doubt that the performance would have been as good. Tests show that the CPUs are being used at close to 100% with occasional small dips for what, from profiling, appears to be garbage collection. The shape of CPU usage on the Ubuntu system monitor is an almost vertical rise to 100% and then a flat line for all CPUs until there is a vertical drop back to normal usage at program completion.
So, Go was so efficient that a Thinkpad X201 almost burned down. I wish I had problems like this with other languages.