Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work.

Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way.

Join us to understand the challenges we faced during this process and how we resolved these challenges.

 
6 favorite thumb_down thumb_up 1 comment visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  1. Quick intro to SAS Platform (5 Mins)
  2. How to unit test ETL and Analytics code in SAS (15 Mins)
  3. How to structure your SAS code (5 Mins)
  4. How to design your modules in SAS (5 Mins)
  5. How to set up a Continuous Integration builds for SAS (10 Mins)
  6. Q & A (5 mins)

Learning Outcome

  1. Understand how to work with Languages/Platforms which have limited support for XP Practices 
  2. Learn how to unit test ETL and Data Analytics Code (in SAS) using SASUnit
  3. Understand how to structure and design your ETL modules for better maintainability and testability
  4. Learn how to setup CI on ETL projects

Target Audience

Teams wanting to apply XP Practices to ETL or Data Analytics Projects. Or Teams implementing XP Practices in new Tech Stack.

schedule Submitted 3 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Rahul Saraf
    By Rahul Saraf  ~  3 years ago
    reply Reply

    Good 1 :)


  • 90 mins
    Workshop
    Beginner

    "To know, is good. To live, is better. To be, that is perfect." - The Mother

    During the Agile adoption, its a common complain that many team in many organizations get caught up in the ceremonies or mechanics of Agile and fail to understand/appreciate the true value and spirit of Agile. And because of this, the original intent of the Agile movement itself is lost. This is a serious issue!

    This workshop will highlight, a well-proven approach to transformation (not adoption) and show the distinct steps in this journey that an individual or a collective goes through when learning anything new. Activities, serving as examples, in the workshop, will focus to show the journey - that is, how to begin with rituals, then gradually move to practices, arriving at principles and eventually internalizing the values. Witnessing this gradual process of transformation will help participants discover for themselves their current progression. We hope this will serve as a guiding light during their Agile journey.

    Finally, we will leave the participants to ponder upon and discover for themselves their ideals in life and work as this is not only applicable to software development, but also to any discipline where humans are involved, including life itself.

  • Liked Sachin Natu
    keyboard_arrow_down

    Sachin Natu - Inverting Test Pyramid - A First Hand Experience Report

    45 mins
    Experience Report
    Intermediate

    Test automation is extremely crucial in adoption of an agile delivery. However, it can take one for a ride, if the approach is not correct. In this sensational, heart throbbing, experience report, we'll share our story of how we turned around an inefficient, expensive automation style to lean, efficient style. In addition to sharing a real-world example, we'll also share some of the key challenges we faced and how we solved them. If you are convinced about the Testing Pyramid, but are struggling to invert it, then this session is for you.

    Business Impact:

      Earlier Defect Detection - Higher test coverage at Unit/Intermediate layers lead to earlier defect detection. Reduced number of issues found on higher test environments/Production. Reduced cost of defect fixing.

      Reduced maintenance cost - UI tests are fragile and costlier to maintain Vs backend tests. No of changes in services layer are comparatively less.

      Reduced test execution time - Backend tests are much faster. Almost 7-10  times faster than UI Tests - improved build certification time.

      Test feedbacks are naturally distributed across layers of application. Test feedbacks are more pin pointed/ granular.

  • Liked Ashish Parkhi
    keyboard_arrow_down

    Ashish Parkhi - Techniques to Speed Up your Build Pipeline for Faster Feedback.

    45 mins
    Experience Report
    Intermediate

    I would like to share my experience and journey on how we brought down our Jenkins build pipeline time down from over 90 minutes to under 12 minutes. In the process, I would share specific techniques which helped and also some, which logically made sense, but actually did not help. If your team is trying to optimize their build times, then this session might give you some ideas on how to approach the problem.

    Development Impact - For one of our build job, below graph shows how the number of builds in a day have increased over a period of time as the build time has reduced. Frequency of code check-in has increased; Wait time has reduced; failed test case faster to isolate and fix.

    Business Impact - More builds leading to quicker feedback and faster story acceptance and less story spill over.

  • Liked Howard Deiner
    keyboard_arrow_down

    Howard Deiner - Agility at Scale - Platform versus Product Concerns

    45 mins
    Talk
    Intermediate

    A common failure mode for organizations attempting to adopt an Agile style of software development occurs when an attempt is made to “Scale Agile”. Suddenly, the organization finds that there are scheduling problems between teams. Delivery team members suddenly find that they are required to serve on several teams at once. Dependencies surface, and teams find it difficult to come together in a common cadence to produce working software in a continuously delivered fashion. Many times, these issues become so grave that the organization reverts back to the Waterfall model that they came to hate, but at least understood.

    This session explores Agile scaling concerns, and places particular emphasis on an architecturally significant distinction in the software to be created, and the components produced to allow the software to be created. That distinction revolves around cross cutting platform concerns versus product feature creation concerns. We will examine the distinctions and explore solutions that should help your organization get past these issues when it comes to portfolio management, by paying attention to extrinsic versus intrinsic value metrics.

  • Liked srinivas chillara
    keyboard_arrow_down

    srinivas chillara - Architectural hiccups (and you really shouldn't bother with sprint-0)

    45 mins
    Experience Report
    Intermediate

    While design changes can be made as we go, sprint on sprint, the poping up of architectural changes seems insurmountable. It is need not be so.

    In this session, we provide a narrative of an online gaming product developed. This is a massively multi-player online game, for casual as well as professional players (with real sustantial money). Such a product needs to grapple with non-functional imperatives well beyond the run of the mill server-side software. As the project progressed new performance and security imperatives materialised.

    How the  team made adjustments in face of changing non-functional (and some perculiar functional) imperatives is the focus of this talk.

  • Liked Alexey Ilyichev
    keyboard_arrow_down

    Alexey Ilyichev - Skype goes agile: don't repeat our mistakes

    Alexey Ilyichev
    Alexey Ilyichev
    Agile Coach
    ScrumTrek
    schedule 3 years ago
    Sold Out!
    45 mins
    Case Study
    Intermediate

    In 2011, I worked for Qik, a startup that got aqcuired by Skype. At that time Skype was in the middle of an agile transition. Аfter aquisition, Qik team was told to adopt the Agile process used by Skype. I worked with the team as an agile coach. After adopting Skype's "agile" process, our ability to deliver was brought down to almost zero. In this talk, I'll tell you the story as it happened, analyze the key problems that we faced and describe how we finally solved them. Come to this talk, if you want to avoid similar mistakes. If you are already through with your transition, I would be interested to know if you see any patterns. 

  • Liked Shirish Padalkar
    keyboard_arrow_down

    Shirish Padalkar - Application Security - The Agile Way

    Shirish Padalkar
    Shirish Padalkar
    Senior Consultant
    ThoughtWorks
    schedule 3 years ago
    Sold Out!
    45 mins
    Talk
    Beginner

    Traditionally application security has involved upfront design and a big bang penetration test after development. This leads to the phenomenon of “bolt-on” security that translates into increased cost and complexity.

    Drawing on our experience on real-world projects, we show how security can be baked-in on an agile project. Using case studies we demonstrate how security concerns are captured during project inceptions, how developers write secure code, security testing is automated and how configuration management can help achieve secure deployments. This talk introduces several new concepts like secure by design, secure design patterns and lightweight code reviews.

  • Liked Howard Deiner
    keyboard_arrow_down

    Howard Deiner - Lean Thinking and What It Mean to the Agile Mindset

    45 mins
    Talk
    Intermediate

    Long before the Agile revolution for software development began, industry had learned that efficient production of goods required intense attention to quality, teamwork, and continuous improvement. These themes of Lean Manufacturing (which was further refined into the Toyota Production System) were never part of the original formulation of the Agile Manifesto, and are rarely mentioned as part of the traditional Agile/Scrum recipe for teams transforming to the new “Agile” mindset.

    The reality is that the traditional Agile/Scrum recipe is actually a “dumbed down” version of the Toyota Production System, and makes it easier for organisations to grasp and start from. However, if organisations really want to achieve the goal of producing the software they need in a fashion that leads to High Performance Teams and Sustainable Engineering, they will need to understand the principles of Lean so they can incorporate them into their unique process. This session teaches the basics of Lean, and demonstrates how they apply to Agile development.

  • Liked Nilotpal Das
    keyboard_arrow_down

    Nilotpal Das - Head First Agile and Organizational Transformation

    45 mins
    Experience Report
    Intermediate

    This is a collection of real time case studies of failed projects. It is an autopsy of these failed projects studying why these projects failed and how application of agie principles could have saved them.

    It is also a study into the organizational culture, behavioral attributes and the people issues and how agile addresses them.

    And finally it is a study of why doing a few projects with agile is not sufficient. How complete organizational transformation into agile practices is necessary for long term success for projects, processes and people.

  • Liked Prafulla Girgaonkar
    keyboard_arrow_down

    Prafulla Girgaonkar - The Art of SQL Database Refactoring

    45 mins
    Experience Report
    Intermediate

    "We've tested this feature thoroughly and it worked really well. But for some weird reason, it's really slow in production today...must be a network issue...or may be the server is having a bad day..."

    Do you often hear these kinds of comments in your development team? Let us guess, your application is very data-centric and churns big blocks of data on every user request. And under the hood, your application is most probably heavily dependent on long/complex queries with joins, temp-tables, case-statements, nested queries, etc.

    These SQL queries probably started-out very simple. But as your requirements evolved, iteration after iterations, the queries also grew in complexity. And most often, even if you test-drove your newer stories, the performance of these complex queries is not evident until you run them in production. 

    Given that our requirements will evolve and so will our database, how do you deal with the above problems?

    There are TWO essential parts to evolutionary database design:

    1. The art of refactoring your SQL queries.
    2. Figuring out the right balance of what processing is done in SQL on the DB sides and what is done on your service side in your App/Web Server.

    Join us as we take a tour of how we refactored our complex, non-performant queries and overall DB without hurting our time-to-market.

  • Liked srinivas chillara
    keyboard_arrow_down

    srinivas chillara - Save our Ship, less is more

    20 mins
    Talk
    Advanced

    Scrum of Scrums has a mixed record of effectiveness. The vast majority completly miss the point of stand-ups and Scrum of Scrums. The last few years have seen unscrupulous consultants sell all sorts of pills for the ills of scaling. If a project needs more than 15 people, it is crucial one understands the Central issue of scaling. 

    In this talk I take the audience through the common pitfalls and explain the irrelavance of the question: "How can we scale agile?"