This is a guest post from Matt Fleming from Code Blueprint. Matt is a Senior Performance Engineer at SUSE.

Some development problems are too complex, some timelines too tight, and some projects too greenfield for established teams to tackle. When you need to create a new team of developers for an ambitious project, the venerable cross-functional or tiger team provides the perfect model for bringing a ragtag crew together to achieve a shared goal.

I had the honor of leading such a team at SUSE. The team’s objective was to optimize the performance of running in-memory databases in a virtualized environment. We were given a testsuite with over 500 SQL performance tests and challenged to make sure the majority of results were no worse than 12% of the bare metal score. When we started out, the virtualized workload performance was over 50% worse. Within six months, we’d achieved our 12% target by carefully tuning and optimizing our system configurations.

Though the team was made up of experts from a variety of areas, the real secret to the project’s success was using automation and monitoring to make sure everyone was analyzing performance the same way.

That’s a lesson we had to learn the hard way: when we started out, each of us ran tests on our own machines using a common set of testing scripts that took care of running the tests but not configuring the machine, monitoring performance metrics, or comparing results. Problems quickly sprang up when we shared our tunings and optimizations with the rest of the team and even the familiar line “it works on my machine” didn’t help with the fact that no one could reproduce the results.

This happened time after time until eventually we agreed to take a step back and implement automation for everything from configuring the machine to figuring out whether performance improved or not. Implementing automation resulted in a step-change in the project’s progress and in our team’s morale.

Below are the lessons we learned on our way to becoming a high-performance team.

#sql