Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work.

Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way.

Join us to understand the challenges we faced during this process and how we resolved these challenges.

"We've tested this feature thoroughly and it worked really well. But for some weird reason, it's really slow in production today...must be a network issue...or may be the server is having a bad day..."

Do you often hear these kinds of comments in your development team? Let us guess, your application is very data-centric and churns big blocks of data on every user request. And under the hood, your application is most probably heavily dependent on long/complex queries with joins, temp-tables, case-statements, nested queries, etc.

These SQL queries probably started-out very simple. But as your requirements evolved, iteration after iterations, the queries also grew in complexity. And most often, even if you test-drove your newer stories, the performance of these complex queries is not evident until you run them in production. 

Given that our requirements will evolve and so will our database, how do you deal with the above problems?

There are TWO essential parts to evolutionary database design:

1. The art of refactoring your SQL queries.
2. Figuring out the right balance of what processing is done in SQL on the DB sides and what is done on your service side in your App/Web Server.

Join us as we take a tour of how we refactored our complex, non-performant queries and overall DB without hurting our time-to-market.