Techniques to Speed Up your Build Pipeline for Faster Feedback.
I would like to share my experience and journey on how we brought down our Jenkins build pipeline time down from over 90 minutes to under 12 minutes. In the process, I would share specific techniques which helped and also some, which logically made sense, but actually did not help. If your team is trying to optimize their build times, then this session might give you some ideas on how to approach the problem.
Development Impact - For one of our build job, below graph shows how the number of builds in a day have increased over a period of time as the build time has reduced. Frequency of code check-in has increased; Wait time has reduced; failed test case faster to isolate and fix.
Business Impact - More builds leading to quicker feedback and faster story acceptance and less story spill over.
Outline/structure of the Session
- My introduction (1 min).
- Why long running pipeline was hurting (2 min)
- Demonstrate example of Disk IO intensive operations ( 2 min)
- How the JUnit ant task options affect the time required to run tests (5 minutes)
- In-memory/In-Process db (5 min)
- SSD, Hybrid Disks, RAM Drives, tmpfs (5 min)
- How CPU profiling gives further insights into our code (10 min)
- CPU intensive operations – e.g. addDays.
- Disk IO intensive operations
- Scanning of resource bundle files.
- Loading Spring context from tests all tests.
- How profiling can bring down not only the time of JUnit tests, but can also have a big impact on your production code efficiency.
- Questions ( 5 min )
- Importance of getting fast feedback.
- How to investigate long running tests.How to profile long running tests using Eclipse.
- How to run Tests concurrently.
- RAM Disks, SSD, Hybrid disks.
- Don’t assume; validate your hypothesis.
Software Developers, Testers, DevOps
schedule Submitted 2 years ago
People who liked this proposal, also liked:
The Tao of Transformation
"To know, is good. To live, is better. To be, that is perfect." - The Mother
During the Agile adoption, its a common complain that many team in many organizations get caught up in the ceremonies or mechanics of Agile and fail to understand/appreciate the true value and spirit of Agile. And because of this, the original intent of the Agile movement itself is lost. This is a serious issue!
This workshop will highlight, a well-proven approach to transformation (not adoption) and show the distinct steps in this journey that an individual or a collective goes through when learning anything new. Activities, serving as examples, in the workshop, will focus to show the journey - that is, how to begin with rituals, then gradually move to practices, arriving at principles and eventually internalizing the values. Witnessing this gradual process of transformation will help participants discover for themselves their current progression. We hope this will serve as a guiding light during their Agile journey.
Finally, we will leave the participants to ponder upon and discover for themselves their ideals in life and work as this is not only applicable to software development, but also to any discipline where humans are involved, including life itself.
Inverting Test Pyramid - A First Hand Experience Report
Test automation is extremely crucial in adoption of an agile delivery. However, it can take one for a ride, if the approach is not correct. In this sensational, heart throbbing, experience report, we'll share our story of how we turned around an inefficient, expensive automation style to lean, efficient style. In addition to sharing a real-world example, we'll also share some of the key challenges we faced and how we solved them. If you are convinced about the Testing Pyramid, but are struggling to invert it, then this session is for you.
Earlier Defect Detection - Higher test coverage at Unit/Intermediate layers lead to earlier defect detection. Reduced number of issues found on higher test environments/Production. Reduced cost of defect fixing.
Reduced maintenance cost - UI tests are fragile and costlier to maintain Vs backend tests. No of changes in services layer are comparatively less.
Reduced test execution time - Backend tests are much faster. Almost 7-10 times faster than UI Tests - improved build certification time.
Test feedbacks are naturally distributed across layers of application. Test feedbacks are more pin pointed/ granular.
Applying Gamification Techniques in Agile
- One of the world’s largest legal product & services provider.
- Development and global rollout of online legal research product for law firms to efficiently provide online research and analytical capabilities.
- Several hundred million dollar product revenue
- 400 Team size.
- Product Development and subsequent enhancement and support.
- Stringent Timelines
- Most of the team new to agile
- 4 Timezones
Key Challenges on "people" front:
- Scale up the team to use agile in short time frame
- Engage and motivate the team
- Be a true vendor partner with the customer
Industry analysts predict rapid adoption of gamification. By 2014, more than 70% of Global 2000 organizations will have at least one gamified application, according to Gartner, Inc. In this presentation, we would be focusing on how gamification helped us adddress the above challenges. Concept of “gamification” was used to drive the team to embrace agile and improve employee engagement and motivation. This helped further boost the quality and increase the overall customer satisfaction across sprints and releases.
eXtreme Programming for ETL and Data Analytics
Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work.
Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way.
Join us to understand the challenges we faced during this process and how we resolved these challenges.
Skype goes agile: don't repeat our mistakesAlexey Ilyichev
schedule 2 years agoSold Out!
In 2011, I worked for Qik, a startup that got aqcuired by Skype. At that time Skype was in the middle of an agile transition. Аfter aquisition, Qik team was told to adopt the Agile process used by Skype. I worked with the team as an agile coach. After adopting Skype's "agile" process, our ability to deliver was brought down to almost zero. In this talk, I'll tell you the story as it happened, analyze the key problems that we faced and describe how we finally solved them. Come to this talk, if you want to avoid similar mistakes. If you are already through with your transition, I would be interested to know if you see any patterns.
Application Security - The Agile WayShirish Padalkar
schedule 2 years agoSold Out!
Traditionally application security has involved upfront design and a big bang penetration test after development. This leads to the phenomenon of “bolt-on” security that translates into increased cost and complexity.
Drawing on our experience on real-world projects, we show how security can be baked-in on an agile project. Using case studies we demonstrate how security concerns are captured during project inceptions, how developers write secure code, security testing is automated and how configuration management can help achieve secure deployments. This talk introduces several new concepts like secure by design, secure design patterns and lightweight code reviews.
Enabling Continuous Delivery (CD) in Enterprises with TestingAnand Bagmar
schedule 2 years agoSold Out!
The key objectives of Organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!
In such a fast moving environment, CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury!
There are various practices that Organizations and Enterprises need to implement to enable CD. Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful.
Testing in Organizations on the CD journey is tricky and requires a lot of discipline, rigor and hard work. In Enterprises, the Testing complexity and challenges increase exponentially.
In this session, I am sharing my vision of the Test Strategy required to make successful the journey of an Enterprise on the path of implementing CD.
Fibonacci Pairing: Learning through CollaborationYilmaz Guleryuz
schedule 2 years agoSold Out!
According to many researches, we learn the best when we experience it personally, and we comprehend when we teach to someone else.
So if you want to achieve ample understanding of a topic, you’ll need to follow these basic steps:
"Listen -> Observe -> Do -> Share"
Pair-programming is a well-known practice, and it has been introduced by one of the thought leaders of Agile world, Kent Beck. I respect any technique that encourages collaboration and interaction among teams, and gets teams to communicate better.
However, if you want to establish understanding of mastery or a skill-set, and increase the shared knowledge across your team members, then you need to re-think & apply new techniques.
In this talk, I’ll share about What-Why-How of "Fibonacci Pairing" technique, which is yet another discovery from practice.
The main goal of that technique is to establish understanding and improve collaboration, without overcrowding by artificial ways like lecturing.
Fibonacci Pairing is simple yet very effective technique that can improve team work. This technique should be regarded as a complementary way to improve team culture effectively.
We will elaborate with practical examples, real life experiences, and applicable ways onto different contexts.
Spotless Clean Frontend Development with BDD, DSL and StoryboardsFrans Overbeek
schedule 2 years agoSold Out!
After 3 years of improving our frontend development process we ended up with mixing BDD, DSL and Storyboards for developing spotless clean frontend code. Each approach is already powerful on his own but combining these will make them unbeatable. The characteristics are collaborate, communicate and visualize, BDD is the collaborating process between disciplines, DSL for enabling the communication and Storyboards for visualization.
The Art of SQL Database Refactoring
"We've tested this feature thoroughly and it worked really well. But for some weird reason, it's really slow in production today...must be a network issue...or may be the server is having a bad day..."
Do you often hear these kinds of comments in your development team? Let us guess, your application is very data-centric and churns big blocks of data on every user request. And under the hood, your application is most probably heavily dependent on long/complex queries with joins, temp-tables, case-statements, nested queries, etc.
These SQL queries probably started-out very simple. But as your requirements evolved, iteration after iterations, the queries also grew in complexity. And most often, even if you test-drove your newer stories, the performance of these complex queries is not evident until you run them in production.
Given that our requirements will evolve and so will our database, how do you deal with the above problems?
There are TWO essential parts to evolutionary database design:
- The art of refactoring your SQL queries.
- Figuring out the right balance of what processing is done in SQL on the DB sides and what is done on your service side in your App/Web Server.
Join us as we take a tour of how we refactored our complex, non-performant queries and overall DB without hurting our time-to-market.