Posts are great – working to make things faster is better

Sorry for the delay in posting (6 months, sweet christ), but I’ve been incredibly absorbed in some great projects we’ve got going on at work recently.

However, we’re at a point of reflection now and I feel like this would be a great time to introduce a couple of concepts that I’ve found are key to making code perform well. These may be basic concepts, but they’re things that are key to reinforce:

  1. The best code you write is code you don’t have to write. While this might sound like an ode to being lazy, it’s not. Think about it in object oriented terms: Write a function you can call instead of several variations on the same code. It hits all the great hallmarks of good code: It ends up being more readable, is more maintainable, and is easier to tune.
  2. Understand your stack and how what you’re writing affects it. When you write a new function, understand what it’s doing and why. Might it be better implemented at a lower level? If you find yourself operating at one layer of the stack, challenge yourself to step up or down and do it differently. Different languages and technologies can inspire a different way of thinking about the solution to a problem.
  3. Don’t be afraid to rewrite something. We always hear about sunk cost and always strive for doing things right the first time, but by making sure that we leave ourselves room to iterate (using branched source control, for one) we let our future self make improve on our own thoughts.
  4. Act like an Engineer, not a developer or code monkey. Apply scientific rigor to what you do: measure, test, and re-measure. Prove out your assumptions using whatever you can (I’m a huge fan of napkin math), and continually think about how to solve problems rather than accomplish tasks.
  5. Never stop learning or being excited. We work in technology, and it’s amazing. From medical science to astrophysics to quantum technology we are seeing breakthroughs every day. The best way I know to keep my motivation up is to continually learn new things, even if it’s outside of my particular job. Learning about how sys-admins accomplish their job led to a better understanding of SAN technology, which leads directly to better tuning it.

These are just a few of the multitude of things I’ve learned over the past few years, and I don’t expect the rate will slow down at this point. Are there things you’ve learned along the way you want to share? Let me know.

Performance in Modern Times: Are we really talking about efficiency?

So I’ve been thinking about what Performance really means and how we generally consider and measure it recently, and wanted to share some thoughts.

A lot of times, we talk about performance in terms of metrics: horsepower or miles per hour or MB/s or response times. However, it is more rare to talk about performance in terms of efficiency and I think that there is an important distinction.

The distinction is that efficiency is a broader term that can encompass a user’s experience (removing a step in a user’s process can yield far more benefits than tuning the underlying queries), hardware (reading less into memory means you can use less memory), programming (the simple concept of reusable code comes into play here), and even documentation (for all we like to blast him, Clippy did guide me to a template now and then, saving me a lot of time).

When we talk about performance or consider how to improve it, I think it’s important that we consider it holistically and methodically. Start with exacting steps and a clear goal in mind (I want to add a user to a database via a web site faster). Understand the problem and what goes into it. Then consider what the best way of improving it is. Sometimes it may not be tuning a query, sometimes it may be improving a process. Sometimes it could just be offering a progress bar so the user feels something is happening.

What do you think?

Performance in Modern Times: What can we do about it?


Spoiler alert: Nope.

Over the past few weeks I’ve talked about performance in very easily understandable terms. I’ve delved into what it is how we currently handle it, even diving into why that doesn’t work. Now it’s the time to  move towards what you can do.

The interesting thing is that there are a lot of layers (like an onion or Dante’s Inferno!), let’s look at the anatomy of a button click in a web-based software with a SQL database backend:

  1. The client clicks a button, which is rendered on their workstation and has information passed along a network to a web server.
  2. The web server receives that request and determines what it needs to do, usually asking the SQL Server for something. It sends a request to the SQL Server.
  3. The SQL Server then receives the request, gathers the data, and sends it back to the web server.
  4. The web server then sends the information to the client workstation.
  5. The client workstation displays the information and the user can move on.

Get all that? Good. Look at what that means in layers that performance problems can stem from:

  • The workstation: CPU overutilization, memory constraints, users not using the right button (a “performance problem” might be user error, after all).
  • Internet: If the workstation is outside of the corporate intranet, the request between the workstation and web server goes across the Internet and there could be any number of variables here (latency, bad switches)
  • Web server: The web server can have all the same constraints as the workstation, with the added possibility of badly configured load balancing or caching.
  • SQL Server: There are so many resources for what can go wrong at this level. At a basic level, the SQL Server can suffer from resource constraints. There are also logical constructs within the database that can contribute to bad performance (bad index design, bad queries)

So how do we tackle this? I talked to a developer about what they thought about Performance tuning. The quote that I remember is “It’s like trying to rub cheetah blood on the server.” and although I suspect the developer was being facetious I think that’s pretty much true. Without the proper frame of mind and resources you’ll be flailing about in the sea of possibilities.

Handle it with Science!

In order to be effective at improving performance we’ve gotta think back to our science classes and remember something very important: scientific methodology.

At a high level, this is creating a testable hypothesis and testing it to reach a conclusion. In the context of software performance, this requires that we control for as many variables as we can, so we need to answer the Who, What, Where, Why and How. I’m going to go into details about each of those, but here’s some basics to get you started:

  • Who: There are specific skills needed to be able to gather information and analyze it to see where the bottlenecks may be.
  • What: Doing science requires having a testable hypothesis, so it’s important to identify what you’re looking for. This helps inform you of things that may impact it (variables in the experimental sense) and allows you to control for those.
  • Where: Specifically in software performance we need to know where we can do the experiments and investigations. This generally means having some sort of isolated environment that we can control.
  • Why: Each of the above things costs us something (time, money, gray hair). We need to constantly be asking ourselves why we are doing this.
  • How: Performance testing is a very specific subset of software testing and there is an entire lexicon around it. Understanding this allows us to not have to reinvent anything, and improves our efficiency while doing experiments.

Keeping these questions in mind will help make you far more effective in tackling performance issues. I’m pretty excited about writing the next few articles as we’ll get to dive into the specifics of answering the above questions.

Performance in Modern Times: How is it normally handled?

It will take HOW MUCH effort to fix?


I always find it interesting to understand where we come from. Whether this be anthropological theory or how a company handles performance bugs, I think understanding the history of how things have been done can help us better plan for the future. In addition, learning the mistakes that others make can help you not make them yourself!

So with that goal in mind, thus begins the next entry into our “Performance in Modern Times” series: How is it normally handled?

During software development, we are often focused on a very specific goal. This could be creating a new piece of functionality, testing functionality, or prioritizing some future work (who has been in meetings about “planning to plan?”). We can get very focused on attaining this goal, and it’s easy to know that we’re solving the right problem because that problem leads directly to that goal.

Add in to this that understanding how a system performs requires an understanding of numerous different layers (for instance: hardware, database, application) and designing efficient tests and metrics for performance requires a considerable amount of effort, and that the classic definition of a piece of functionality being complete means that you can run it successfully, and you start to get the picture that performance testing is:

  • Difficult
  • Costly

And these two things lead to it being lower priority than other tasks. So in a modern development cycle performance might be relegated to a task that gets added on if time exists for it.

This means that while a developer might try their best to make something perform will, they won’t be able to prove it. So no one will know that something is slow until the client is using it and the testing at scale is done during production.

So that leads us to the next obvious question for the development lead or product manager:

Can this work?

Sure! It can work for a given definition of “working”. It will work if you don’t mind clients complaining when something is slow and you have to fix it after the fact. In a world of infinite resources this won’t matter, as you can just get a developer to tune the code after the fact, right?

Sort of. Take a look a the chart at the beginning of the article. It’s a representation of the cost (in amount of work) when a performance issue is discovered. Let’s walk through it and see how quickly the effort can add up:

  • The top part is during design time. This is when you’re sketching out the design on a white board (or a chalk board if you live in the 1950s), and you’re trying to understand what the piece of functionality is. At this point, if you discover a performance issue (for instance, there’s 30 fields that are required to save the data) it only requires erasing the design and rethinking it. The designer is the only one on the hook for this. Maybe instead of each form you prepopulate some fields and obfuscate others behind a drop down.
  • Next comes during the development (read: actually mashing on the keyboard to create code) phase. If a performance bug is found here (for instance saving that form doesn’t return results from the database in less than 1 minute – so slow!) the developer is on the hook for this and will have to rewrite or rethink the code. More effort than what the designer would need to do, but not necessarily by that much.
  • Usually after the developer has some time to write up a prototype or proof of concept, that code will be turned over to QA for validation. At this point if a performance bug is found it’s going to require QA to replicate/narrow down the cause, a developer to look at it and fix it, and then QA to retest it. It’s quickly taken more effort at this point, because more people are involved.
  • Once QA and Dev sign off on this process we’re going to need to package up the feature to be placed into production. This is where the system administrator (note that this is an intentionally vague term, it could be a DBA or a Windows admin or something similar) comes in. Let’s say that at this point they determine that we’ve added 5 more large tables and it’s causing memory usage to increase (with a notable decrease in Page Life Expectancy). So at this point they would need to rope the Development team back in and it would need to be: Replicated by QA in a test environment, fixed by a Developer, retested by QA to ensure it doesn’t persist. Only then would the administrator be able to reinstall the feature and consider the performance bug fixed.
  • Finally, once everything is said and done the client gets to use the feature. Unfortunately for a lot of products this is where things get hairy, because clients will use the feature in incredibly innovative ways and throw all sorts of new scenarios at it. So if we start seeing a web page time out when a client has 5 items in their cart and tries to split payment 3 different ways, we’ll have to involve a lot of people: The client to report the issue, Support to troubleshoot the issue and interact with the client, the system administrator to confirm that the production system wasn’t experiencing an outage, QA to narrow down the cause, a developer to fix it, QA to retest it, a sys admin to reinstall it (or install a patch), and the client to let us know it’s fixed.

Whew! That’s a lot of words about this. I have the suspicion that you might agree that addressing issues early in the process (like during design or development) is more cost effective. The next step is to take a look at how we can effectively handle this within the constraint of a development cycle.

That’s gonna be the next post! I’ll give you a quick spoiler though, since you’ve read this far: It’s something everyone can help with.

Performance in Modern Times: Why is it important?

Whee! Go faster!

No, not that Modern Times. Our current modern times, with robots and all sorts of lasers.

In this series, I’m going to explore what software performance means in our modern world, as well as sharing some insights I’ve had while working within an organization to improve the performance of a database software. This topic is pretty close to all of our hearts, as in our jobs we are all called upon to do things more efficiently.

First off, it’s important to understand why performance is even important. One problem that a lot of organizations will make is that performance is only something you need to consider once something is slow. More proactive organizations might even take the step of testing for performance as part of a regression test before launching the software.

However at this point it might be more costly and risky to address the problems causing performance issues. Take a simple example: Allowing files to be stored in a MS-SQL database. While this might be beneficial for the end user (one interface that allows them to interact with files without leaving it! Awesome!), storing these files in the database rather than the file system means that if those files are heavily referenced, they’re going to reside in memory on the database server. That can mean that other things will be pushed out of memory and onto the slower disk.

To address this problem, you would have to re-engineer the feature to reference files on disk rather than in the database, or take another measure that would take additional development and testing time. If this problem was discovered after the software shipped, you would have to also consider backwards compatibility, upgrade paths and possibly even conversions. Not fun.

However, if the performance impact of this feature had been considered when the design was being discussed (“Hey, wouldn’t these files be pulled into memory on the server when they’re being used?”) alternatives could have been explored and a fix would have been unnecessary. This could save money in the long run, time, and contribute to overall code health. One thing I like to do is constantly ask myself, “Am I using the right tool for the problem I’m trying to solve?” and more often than not, it leads me to explore different ideas whether this be an alternative database structure or even just a different way of monitoring performance on one of our servers.

Performance is pervasive, as well. Everywhere we look, things are going so fast it’s insane. Consider that smart phone you’re looking at. It’s transferring millions of bits a second (4G has a standard of 100 megabits per second for mobile devices) and transforming those bits into things you can use. Like a weather report or this blog. We’re used to being able to turn on the TV and instantly get moving images with sound (requiring much complex engineering even though it’s an old technology at this point). Your clients (remember those, from the last article?) are simply expecting as a baseline nearly instant access to information (after all, I can search the entire internet in less than half a second, why can’t I get information from the database that fast?): they will only notice performance if it is bad. This makes it even more important to be asking those questions early in the process!

Stay tuned for the next post, we’re going to be exploring the various aspects of how performance is currently handled and tuned.

Devs hate him! Learn this analyst’s one weird trick to improve performance!

Performance Engineering is a weird and wonderful thing. You get to learn about almost every part of technology as part of your job, and more than that you get to actually use that knowledge to make things go faster.

In this blog I’m going to share some of the things I’ve learned from working for a software vendor solutions provider, notably in the area of making things run more efficiently and quickly (which, by the way, should probably be rolled up into “working”, but more on that later). Why should you care about what I have to say? Well, in my position I am extremely lucky in that I have insight into many different aspects of the product lifecycle and am equal parts consultant, server admin, database admin, SQL developer, .NET developer and product manager. So while I love technology and all we do with it, my job is focused on the practical application of that technology: how to improve performance.

That’s enough about me though. I have a lot of general knowledge, and love sharing. So instead of just making this a normal intro post that will leave you wanting, let’s dive in and take a look at something I bet a lot of you know intrinsically but may have not considered. It’s a key lesson I’ve had to learn:

Our clients are not always paying us

At least, not directly! As a Performance Engineer you may not have a job where you face a client who pays directly for your services. In my position, our clients are loosely defined as the people who buy and use our software. Take a mental step back (or a physical step back if you are reading this on the train and are so enraptured by my words that you don’t notice that guy trying to get past you) and consider what makes a client a client.

A client is someone who consumes the work you do. So it’s someone who benefits (hopefully) from things you do. In that sense, there’s a lot of clients we have to consider:

  • If you’re a DBA, your clients include internal users of the databases you manage and executives who read reports from those
  • If you’re a System Administrator, your clients include every user who uses the system you manage
  • If you’re a process manager (a Product Manager or Program Manager, for instance), your clients are those who use those processes

With this in mind, we can better understand what we have to do to really make our clients like us (and their approval is what we desire with every waking breath, right?). We need to put ourselves in their shoes, so that we can better understand what their concerns might be. Engage them (internal clients are easy, you can just email them or walk over and talk to them!) for feedback often, and try to understand their priorities.

I’ll close with an anecdote that illustrates an idea on how to make clients happy:

When I worked in Product Support, our clients were mainly the people who used our software on a day to day basis. I would talk to them every single day and endeavor to help them any way I could (we have fantastic clients, you can’t help but want to help them and go the extra mile). When we were working on a particularly difficult issue that required we take time to troubleshoot and research the issue on our own, it was encouraged that we reach out to the client every other day with an update.

This metric did a few things that were important:

  1. Helped keep us focused and organized, as you never wanted to reach out to a client with “Hey, no new news.”
  2. Reminded the client that we understood their priorities and didn’t forget about them
  3. Improved overall communication between Support and the client

I never noticed how much of a difference this made until I was on the other side of one of these problems, with a difficult problem that I needed help with. The person who was working on my problem was working diligently but reached out with regularity to let me know it hadn’t escaped their attention. Those little two line emails helped me to organize my day better and appreciate the gravity of the problem. That person understood I was sitting there and in a dearth of updates I would probably be concerned if things were ever going to move or get fixed.

Because of those updates, I was very satisfied and able to continue working. So take a moment to think about who your clients are and what you can do to make their days a little better. It might just take a quick email letting them know you’re still working on a longer problem!