Blazing fast Big Data processing - High performance big data processing using Micro-services & Flow Based Programming

An experience share about how we achieved high performance transaction pricing with elastic scaling, for a big T in the US, using Micro-services and Flow Based Programming.

We start with the challenges faced by the telecom operator in their All IP access network which necessitates high volume, low latency transaction processing.

We discuss the Micro-services advantage which let us build responsive, resilient and elastic services.

We further discuss the FBP edge, which help us manage the big data effectively and also work as an event driven framework for Micro-services

We will have a short discussion on the software stack used.

We conclude by sharing the results and the lessons learnt

 
 

Outline/Structure of the Experience Report

An experience share about how we achieved high performance transaction pricing with elastic scaling, for a big T in the US, using Micro-services and Flow Based Programming.

We start with the challenges faced by the telecom operator in their All IP access network which necessitates high volume, low latency transaction processing. - 3 minutes

We discuss the Micro-services advantage which let us build responsive, resilient and elastic services. - 5 minutes

We further discuss the FBP edge, which help us manage the big data effectively and also work as an event driven framework for Micro-services - 5 minutes

We will have a short discussion on the software stack used - 3 minutes

We conclude by sharing the results and the lessons learnt - 3 minutes

Learning Outcome

Understand how Micro-services architecture and flow based programming work together to build responsive, resilient and elastic systems

Target Audience

Software Architects, programmers

schedule Submitted 11 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • Vivek Ganesan
    By Vivek Ganesan  ~  10 months ago
    reply Reply

    Thanks for the submission, Rajith :)

    I like these aspects of the submission, and they should be retained:

    • Focus on resiliency
    • Sharing Actual experiences

    I think the submission could be improved by:

    • Connecting the dots on how this helped agility, or not 
    • Maybe, a short part about what is flow based programming.
    • Deepti Tomar
      By Deepti Tomar  ~  11 months ago
      reply Reply

      Hello Rajith,

      Thank you for submitting the proposal on "Blazing fast Big Data processing - High performance big data processing using Micro-services & Flow Based Programming". Thanks for your time and efforts on providing a detailed abstract.

      The Program Committee would require some more information from you to be able to review your proposal for all its aspects. It would be great if you could provide us with the link to Slides and Videos to your Previous Conferences/Presentations and Time - Break up on each sub - topic under Outline/Structure through the proposal form here. 

      In order to confirm completeness of the proposal, I'd suggest you to go through the following link to Overview of Review Process - https://confengine.com/review-process#selection-criteria

      We'll certainly keep you updated in case we need more information once you have updated your proposal for the required information through the relevant fields and as per the points mentioned in Overview of Review Process.

      Please let us know in case of doubts/queries,

      Thanks,

      Deepti

      • Rajith Raveendranath
        By Rajith Raveendranath  ~  11 months ago
        reply Reply

        Thanks for the review. I have shared the slides from my session at Agile Gurugram this year. As to the time break up, it's a 20 minutes session and I would be talking about the challenge, the Microservices architecture and the FBP design for 5 minutes each. Last 5 minutes would be to wrap up 

        • Deepti Tomar
          By Deepti Tomar  ~  11 months ago
          reply Reply

          Dear Rajith,

          Thanks for your response and slide update on proposal. It's really helpful.

          We request you to update the Time - Break up on proposal as well. This would be really helpful to the reviewers and for maintaining consistency across all the proposals.

          It would be great if you could provide us a link to the video(s) of your past sessions/conferences. Would you have it available for Agile Gurugram ? Request you to update the same on the proposal too.

          Please let us know in case of any concerns,

           

          Thanks,

          Deepti


      • Tobias Anderberg
        Tobias Anderberg
        Developer/Coach
        Agical AB
        schedule 11 months ago
        Sold Out!
        45 Mins
        Talk
        Intermediate

        Ever wondered why some people prefer to work alone? Or why some people cringe when pair programming is mentioned? It might be that that person, like me, is an introvert. But is is really that simple? Can we really put every person in a box labeled "introvert" or "extrovert" or are we all just ambiverts?

        During this session I will talk about introverts, extroverts and everything in between.
        Drawing from almost 15 years of personal experience being an introvert on agile teams I will talk about the differences of being an extrovert
        or an introvert, how to foster an inclusive team environment, and the importance of psychological safety.
        You will hopefully leave this session better fit to help EVERYONE on your team to reach their full potential!

      • Liked Vincent FUCHS
        keyboard_arrow_down

        Vincent FUCHS - Keeping hundreds of code repositories consistent, and staying sane !

        Vincent FUCHS
        Vincent FUCHS
        Technical architect
        Societe Generale
        schedule 11 months ago
        Sold Out!
        20 Mins
        Demonstration
        Intermediate

        With the move to microservices architecture, a lot of teams end up managing dozens of code repositories (vs just a couple before), and some tasks that were done quickly manually are now becoming very time consuming : consistency of the repositories, and eventually of your platform, gets impacted, making it more and more difficult to manage.

        Surely, there must be some tools existing to take care of boring tasks like finding where a given dependency is used, and upgrade it automatically (this is just an example)... Well actually, we didn't find anything, so we implemented them ourselves and made them available for everyone !

      • Liked Gabor Devenyi
        keyboard_arrow_down

        Gabor Devenyi / Alex Sloley - The magic number is 10

        45 Mins
        Talk
        Beginner

        Why are Agile teams supposed to be small? How big are they supposed to be? Most agilists tend to agree that a team of ten people works well.

        But what is it about the number 10 that makes it the “magic” number?

        Since the start of human evolution, people formed groups to be more effective. Whether it was the hunt for a mammoth or going to war, working in teams ensured a greater chance of success.

        There have been various researches from Dunbar’s paper through the Scrum Guide to military formations about the ideal number of people in a team.

        We’ll discuss the historical, scientific and cultural reasons why 10 seems to be the magic number of forming effective teams.

        Does the number of team members really matter? Is 10 really the magic number. You will get an answer that will help you to create effective teams with the right amount of people.

      • Liked Vivek Ganesan
        keyboard_arrow_down

        Vivek Ganesan / Rejikrishnan Rajan - Reinventing DevOps Practices for Blockchain Applications

        90 Mins
        Workshop
        Intermediate

        With the arrival of distributed applications that run on Blockchain, it is essential to rethink some of the basic aspects of DevOps. Few things that we take for granted in DevOps area are either irrelevant, impossible or too costly for blockchain applications. Join us to explore the changes that the blockchain apps bring to the DevOps arena, using a hands-on demo of a sample blockchain app.

      • Liked Pradeep Kellangere
        keyboard_arrow_down

        Pradeep Kellangere / Zaheerabbas Contractor - Implementing Rapid and Repeatable Pipeline for an Enterprise

        45 Mins
        Case Study
        Intermediate

        Enterprises and their individual feature teams are in a rush to build DevOps Pipelines with all the right reasons.

        While central teams are provisioning tools to be consumed at Enterprise level, individual feature teams are focusing heavily on building pipelines instead of building and enhancing business priorities.

        This results in "duplicate pipelines” solving the same problem of automation for build, test, package, infra, deploy and release using the given toolsets. - This results into duplication of efforts and redundancy.

        We intend to share our approach to build a pipeline enabled to for polygot programming languages and consumption patterns supporting various areas including.

        • A single click to deploy the complete pipeline infrastructure and app pipeline
        • Support multiple cloud deployment (AWS, Azure, etc…)
        • Adoption of open source tools
        • Clustered or non-clustered infrastructure deployments

        We will also emphasize on the implementation experience of the advanced Engineering practices in the area of immutable infrastructure, Chaos Engineering, blue-green/canary deployment.

      • Liked Jen Krieger
        keyboard_arrow_down

        Jen Krieger - So you want to be a DevOps Engineer?

        Jen Krieger
        Jen Krieger
        Chief Agile Architect
        Red Hat
        schedule 11 months ago
        Sold Out!
        45 Mins
        Talk
        Beginner

        You’ve worked hard, you have mastered every coding language on the planet, including COBOL. You know jenkins, travis, go, containers, puppet, chef, CFengine, nagios, github, graphite, logstash, ansible, aws, gce, vagrant, cms, cvs, abc, 123, and even a little bit of TFS. Your resume is up to date, and you are ready for your next job.

        Well, slow down there… Industry trends in engineering are showing a growing desire in companies to hire people who have proven soft skills, are good at collaborating with others, and can regularly solve the most complex problem we face today: effectively talking to our fellow humans.

        Please join Jen for a self-effacing and situationally humorous conversation about the journey of becoming more comfortable with collaboration, open communication and reaching through “the wall.”

        Topics will include:

        • “Collaborate or Die” survival skills
        • The dreaded “soft skills” and how to become comfortable with them
        • What to do in common situations all engineers face
        • How to convince others that your idea is the right idea
        • How do you get the time you need to get your work done