Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work.

Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way.

Join us to understand the challenges we faced during this process and how we resolved these challenges.

 
 

Outline/Structure of the Experience Report

  1. Quick intro to SAS Platform (5 Mins)
  2. How to unit test ETL and Analytics code in SAS (15 Mins)
  3. How to structure your SAS code (5 Mins)
  4. How to design your modules in SAS (5 Mins)
  5. How to set up a Continuous Integration builds for SAS (10 Mins)
  6. Q & A (5 mins)

Learning Outcome

  1. Understand how to work with Languages/Platforms which have limited support for XP Practices 
  2. Learn how to unit test ETL and Data Analytics Code (in SAS) using SASUnit
  3. Understand how to structure and design your ETL modules for better maintainability and testability
  4. Learn how to setup CI on ETL projects

Target Audience

Teams wanting to apply XP Practices to ETL or Data Analytics Projects. Or Teams implementing XP Practices in new Tech Stack.

schedule Submitted 5 years ago

Public Feedback


    • Liked Ashish Parkhi
      keyboard_arrow_down

      Ashish Parkhi / Naresh Jain - Techniques to Speed Up your Build Pipeline for Faster Feedback.

      45 Mins
      Experience Report
      Intermediate

      I would like to share my experience and journey on how we brought down our Jenkins build pipeline time down from over 90 minutes to under 12 minutes. In the process, I would share specific techniques which helped and also some, which logically made sense, but actually did not help. If your team is trying to optimize their build times, then this session might give you some ideas on how to approach the problem.

      Development Impact - For one of our build job, below graph shows how the number of builds in a day have increased over a period of time as the build time has reduced. Frequency of code check-in has increased; Wait time has reduced; failed test case faster to isolate and fix.

      Business Impact - More builds leading to quicker feedback and faster story acceptance and less story spill over.

    • Liked Dhaval Dalal
      keyboard_arrow_down

      Dhaval Dalal / Naresh Jain - The Tao of Transformation

      90 Mins
      Workshop
      Beginner

      "To know, is good. To live, is better. To be, that is perfect." - The Mother

      During the Agile adoption, its a common complain that many team in many organizations get caught up in the ceremonies or mechanics of Agile and fail to understand/appreciate the true value and spirit of Agile. And because of this, the original intent of the Agile movement itself is lost. This is a serious issue!

      This workshop will highlight, a well-proven approach to transformation (not adoption) and show the distinct steps in this journey that an individual or a collective goes through when learning anything new. Activities, serving as examples, in the workshop, will focus to show the journey - that is, how to begin with rituals, then gradually move to practices, arriving at principles and eventually internalizing the values. Witnessing this gradual process of transformation will help participants discover for themselves their current progression. We hope this will serve as a guiding light during their Agile journey.

      Finally, we will leave the participants to ponder upon and discover for themselves their ideals in life and work as this is not only applicable to software development, but also to any discipline where humans are involved, including life itself.

    • Liked Sachin Natu
      keyboard_arrow_down

      Sachin Natu / Naresh Jain - Inverting Test Pyramid - A First Hand Experience Report

      45 Mins
      Experience Report
      Intermediate

      Test automation is extremely crucial in adoption of an agile delivery. However, it can take one for a ride, if the approach is not correct. In this sensational, heart throbbing, experience report, we'll share our story of how we turned around an inefficient, expensive automation style to lean, efficient style. In addition to sharing a real-world example, we'll also share some of the key challenges we faced and how we solved them. If you are convinced about the Testing Pyramid, but are struggling to invert it, then this session is for you.

      Business Impact:

        Earlier Defect Detection - Higher test coverage at Unit/Intermediate layers lead to earlier defect detection. Reduced number of issues found on higher test environments/Production. Reduced cost of defect fixing.

        Reduced maintenance cost - UI tests are fragile and costlier to maintain Vs backend tests. No of changes in services layer are comparatively less.

        Reduced test execution time - Backend tests are much faster. Almost 7-10  times faster than UI Tests - improved build certification time.

        Test feedbacks are naturally distributed across layers of application. Test feedbacks are more pin pointed/ granular.

    • Liked Howard Deiner
      keyboard_arrow_down

      Howard Deiner - Agility at Scale - Platform versus Product Concerns

      45 Mins
      Talk
      Intermediate

      A common failure mode for organizations attempting to adopt an Agile style of software development occurs when an attempt is made to “Scale Agile”. Suddenly, the organization finds that there are scheduling problems between teams. Delivery team members suddenly find that they are required to serve on several teams at once. Dependencies surface, and teams find it difficult to come together in a common cadence to produce working software in a continuously delivered fashion. Many times, these issues become so grave that the organization reverts back to the Waterfall model that they came to hate, but at least understood.

      This session explores Agile scaling concerns, and places particular emphasis on an architecturally significant distinction in the software to be created, and the components produced to allow the software to be created. That distinction revolves around cross cutting platform concerns versus product feature creation concerns. We will examine the distinctions and explore solutions that should help your organization get past these issues when it comes to portfolio management, by paying attention to extrinsic versus intrinsic value metrics.

    • Liked srinivas chillara
      keyboard_arrow_down

      srinivas chillara - Architectural hiccups (and you really shouldn't bother with sprint-0)

      45 Mins
      Experience Report
      Intermediate

      While design changes can be made as we go, sprint on sprint, the poping up of architectural changes seems insurmountable. It is need not be so.

      In this session, we provide a narrative of an online gaming product developed. This is a massively multi-player online game, for casual as well as professional players (with real sustantial money). Such a product needs to grapple with non-functional imperatives well beyond the run of the mill server-side software. As the project progressed new performance and security imperatives materialised.

      How the  team made adjustments in face of changing non-functional (and some perculiar functional) imperatives is the focus of this talk.

    • Liked Alexey Ilyichev
      keyboard_arrow_down

      Alexey Ilyichev - Skype goes agile: don't repeat our mistakes

      Alexey Ilyichev
      Alexey Ilyichev
      Agile Coach
      ScrumTrek
      schedule 5 years ago
      Sold Out!
      45 Mins
      Case Study
      Intermediate

      In 2011, I worked for Qik, a startup that got aqcuired by Skype. At that time Skype was in the middle of an agile transition. Аfter aquisition, Qik team was told to adopt the Agile process used by Skype. I worked with the team as an agile coach. After adopting Skype's "agile" process, our ability to deliver was brought down to almost zero. In this talk, I'll tell you the story as it happened, analyze the key problems that we faced and describe how we finally solved them. Come to this talk, if you want to avoid similar mistakes. If you are already through with your transition, I would be interested to know if you see any patterns. 

    • Liked Shirish Padalkar
      keyboard_arrow_down

      Shirish Padalkar - Application Security - The Agile Way

      Shirish Padalkar
      Shirish Padalkar
      Lead Consultant
      ThoughtWorks
      schedule 5 years ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Traditionally application security has involved upfront design and a big bang penetration test after development. This leads to the phenomenon of “bolt-on” security that translates into increased cost and complexity.

      Drawing on our experience on real-world projects, we show how security can be baked-in on an agile project. Using case studies we demonstrate how security concerns are captured during project inceptions, how developers write secure code, security testing is automated and how configuration management can help achieve secure deployments. This talk introduces several new concepts like secure by design, secure design patterns and lightweight code reviews.

    • Liked Howard Deiner
      keyboard_arrow_down

      Howard Deiner - Lean Thinking and What It Mean to the Agile Mindset

      45 Mins
      Talk
      Intermediate

      Long before the Agile revolution for software development began, industry had learned that efficient production of goods required intense attention to quality, teamwork, and continuous improvement. These themes of Lean Manufacturing (which was further refined into the Toyota Production System) were never part of the original formulation of the Agile Manifesto, and are rarely mentioned as part of the traditional Agile/Scrum recipe for teams transforming to the new “Agile” mindset.

      The reality is that the traditional Agile/Scrum recipe is actually a “dumbed down” version of the Toyota Production System, and makes it easier for organisations to grasp and start from. However, if organisations really want to achieve the goal of producing the software they need in a fashion that leads to High Performance Teams and Sustainable Engineering, they will need to understand the principles of Lean so they can incorporate them into their unique process. This session teaches the basics of Lean, and demonstrates how they apply to Agile development.

    • Liked Nilotpal Das
      keyboard_arrow_down

      Nilotpal Das - Head First Agile and Organizational Transformation

      45 Mins
      Experience Report
      Intermediate

      This is a collection of real time case studies of failed projects. It is an autopsy of these failed projects studying why these projects failed and how application of agie principles could have saved them.

      It is also a study into the organizational culture, behavioral attributes and the people issues and how agile addresses them.

      And finally it is a study of why doing a few projects with agile is not sufficient. How complete organizational transformation into agile practices is necessary for long term success for projects, processes and people.

    • Liked Prafulla Girgaonkar
      keyboard_arrow_down

      Prafulla Girgaonkar / Naresh Jain - The Art of SQL Database Refactoring

      45 Mins
      Experience Report
      Intermediate

      "We've tested this feature thoroughly and it worked really well. But for some weird reason, it's really slow in production today...must be a network issue...or may be the server is having a bad day..."

      Do you often hear these kinds of comments in your development team? Let us guess, your application is very data-centric and churns big blocks of data on every user request. And under the hood, your application is most probably heavily dependent on long/complex queries with joins, temp-tables, case-statements, nested queries, etc.

      These SQL queries probably started-out very simple. But as your requirements evolved, iteration after iterations, the queries also grew in complexity. And most often, even if you test-drove your newer stories, the performance of these complex queries is not evident until you run them in production. 

      Given that our requirements will evolve and so will our database, how do you deal with the above problems?

      There are TWO essential parts to evolutionary database design:

      1. The art of refactoring your SQL queries.
      2. Figuring out the right balance of what processing is done in SQL on the DB sides and what is done on your service side in your App/Web Server.

      Join us as we take a tour of how we refactored our complex, non-performant queries and overall DB without hurting our time-to-market.

    • Liked srinivas chillara
      keyboard_arrow_down

      srinivas chillara - Save our Ship, less is more

      20 Mins
      Talk
      Advanced

      Scrum of Scrums has a mixed record of effectiveness. The vast majority completly miss the point of stand-ups and Scrum of Scrums. The last few years have seen unscrupulous consultants sell all sorts of pills for the ills of scaling. If a project needs more than 15 people, it is crucial one understands the Central issue of scaling. 

      In this talk I take the audience through the common pitfalls and explain the irrelavance of the question: "How can we scale agile?"