Archives for November 2011

Nov 2011
AUTHORMike Benner

Deployment, What Really Matters

Do It LiveAt AuthorityLabs we have a saying, “We’ll Do It Live“. This goes to the fact that anyone in the company has the ability to deploy any part of our system to the production environment. There are no keys to the kingdom as it were and it is a rather large kingdom.

Many companies nowadays have this policy and rely on their tests suites and continuous integration systems to catch bugs and failing tests but that to me is the easy part. What we have found over the last year is that it is rarely a bad test or worse a missed test case that causes us problems. It is usually something under performant code or infrastructure issue that pops up at scale. We thinking about our issues we started looking at how we could handle this without taking away the ability for anyone to deploy updates as quickly as possible.

What we found was missing was metric driven deployment. The concept isn’t new, but it is overshadowed with all the talk of CI and continuous deployment. Now when we deploy something, we have a dashboard that show us all of our critical metrics (work throughput, server loads, queue lengths, etc) and these are watched looking for variations that are out of line with the norm and expected. It is amazing what you will see once you hit certain scale thresholds. This system let’s us to a couple of things.

We can rollback the deployment and reevaluate the code, spin up additional resources to make up for the drop, make changes to our infrastructure to deal with things like connection limits or whatever. This has had an interesting and welcome side effect. We now monitor and check many more metrics than before and have a better idea of the health of our system. It also has caused us to think about our code in a different way. We now put more thought into how it is going to affect the system as a whole and immediately add support for any new metric we think will need to be tracked for new features.

Does this mean we have fewer issues? No (we are doing it live remember) but we are able to deal with them faster and in a better way. We aren’t just rolling back after someone complains and then investigating. We are seeing in real-time the actual problems and correcting them were they need to be corrected.

I will follow this up with a post on our tools (Scout , New Relic, etc) and some of the internal things we have done to tie this together. In the meantime sit back and enjoy Mr O’Reilly do it live:

Nov 2011
AUTHORMike Benner



NoSQL Realities

My Twitter stream and usual haunts on the Internet have recently seen an increase in the NoSQL bashing. The one common thread seems to be that “pick your NoSQL” solution is not as good as “pick your SQL” solution at “pick your topic”. I am not here to try debunk these statements or prove one or the other wrong, I would just like us to be comparing apples to apples and having a real conversation about when and where to use the right solutions regardless of the camps they fall in.

First let’s be realistic, NoSQL is not going away and will be more and more a part of our lives everyday, so before taking the fanboy comments on Twitter to heart do yourself a favor and read up on the pros and cons on any solution you are going to use and run some tests on your laptop. Most of the time there is more than one solution that will work for your needs and better understanding the focus and future direction of the technology can help make that decision.

OK, now for the part of the conversation I think is missing:

  1. My NoSQL is more performant than your SQL!
    This statement is not only bold, but very vague. What do you mean more performant? Are we talking about server resources, reads per second, writes per second, etc? Come on this is just going to start an argument where everyone is comparing metrics and benchmarks that are not relavent to each other.Also, you can configure most SQL systems to perform on the levels of their NoSQL counterpart but doing so will degrade their performance in other areas. Doing this maybe beneficial for your team/company in not making them learn a new technology, but also hampers you when leveraging some other feature in your SQL that is not configured correctly anymore.
  2. NoSQL is immature and not ready for production
    This will vary by solution. I would argue that the file system is more mature than any SQL solution (yes it is a NoSQL solution), but I would also say that many of the new kids on the block should be tried and tested before moving them to production and you should expect to have problems and find bugs that have already been worked out in the older, more stable SQL systems. This however is not a reason to dismissed the solution, it is a reason to spend more time reading up on it and talking to the few that are running it so you don’t make the same mistakes
  3. NoSQL can’t do everything SQL can
    Of course it can’t, it isn’t meant to. Most NoSQL systems are built to target a very specific pain point and they accomplish by abandoning features and overhead that most SQL systems implement. This doesn’t mean go implement every NoSQL solution known to man to gain a few milliseconds in your system, but if you find a solution that can make a significant impact on the performance of your application or save you a tremendous amount of time, then it may be time to think about moving that functionality into a NoSQL solution.
  4. NoSQL is not secure
    This is true for a lot of NoSQL solutions. I am not sure why this has been handled this way, but there is good news. You can solve this with your operating system and/or firewall. This is a valid concern and you really need to be aware of how this affects you and your data when implementing any solution.

That is a short list of the statements being flung around, but I think you get the idea.

I don’t know of any NoSQL solution that claims to be a drop in replacement for all things SQL. The performance gains many NoSQL solutions are able to claim come at the expense of not being able to do many of the things SQL can and pushing these concerns out of the database system and back up to the developer. This can be both a blessing and a curse, but with frameworks, ORMs and the such these can be mitigated, but that is a whole other issue that could use some discussion and actually muddies the water even more.

Next time you want to bash or defend NoSQL, think about your reasons, the context and the real world implementations then take the conversation somewhere that allows you more than 140 characters.