Your metrics are showing a decrease in performance and users are starting to complain. You start digging through the performance metrics, source code and your stack to find inefficiencies and places you can gain that performance back. What if you do not find any low hanging fruit and solutions are going to be invasive, complicated and time consuming? While those things may need to be done, do not just dive in and start the surgery, stop and ask who am I doing this for?
Humans are driven by numbers. Why do you think there are some many top 10 lists, 5 easy steps to be a guru, etc? Most developers just start attacking the problem with the goal being to get those numbers back in an acceptable range. Maybe they through up some caching, rewrite some queries or upgrade a server. Ultimately they get the numbers back into that arbitrary acceptable range, wrote some fancy code and might have wasted their time.
Yes, I said wasted their time. We need to step back and look at more than the performance numbers. Look at the analytics for your app. What does the actual user usage look like? How does that data looked when overlaid on your performance metrics? Look for the patterns and correlation of the datasets. The insight you gain might surprise you and point you to where the real problem lies. If you find a rarely used feature(s) seriously consider removing them. You may be surprised how one such feature can bottleneck the system.
Removing a feature that is causing a table lock or extreme CPU spikes can free up those resources allowing the rest of the site to flow and with the usage data you can assess how valuable that feature really is. If you can’t decide one way or the other, ask your users. At the end of the day you may need that feature but your users can live without it for increased performance while you work on a better long term solution.
Don’t be afraid of throwing that hard work in trash. While it may seem like your app is moving backwards what you are really doing is improving your user experience, code and performance maintainability and doing so with the right combination of data to back up this decision.