Operating at scale with Elastic Search in production

schedule Sep 1st 02:55 - 03:15 PM place Grand Ball Room 2 people 54 Interested

Having problems in your application taking hours of effort to debug because of needle in a haystack of errors? Or unable to find anomalies quickly before it booms up to an issue? Want to do behaviorial analysis to learn about your customers? Want to find customers near you? Elasticsearch would be an one stop solution. You can convert data into valuable insights by uncovering hidden patterns and correlations, with its efficient full text search and analytics.

Setting up a cluster in smaller scale is simple. But, complexity grows with scale for any distributed system. Elasticsearch is not far from this fact. This presentation will cover the below:

  • Challenges met during the journey and the learnings that built us more stronger
  • Designing & Building a highly scalable and resilient reactive application to process/ingest multi billions of records/day into elasticsearch
 
1 favorite thumb_down thumb_up 11 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Talk

  • Elasticsearch - Basics.
  • Challenges while scaling application for high throughput.
  • Tuning Elasticsearch for production use.
  • Resilient by design (Application architecture and reactive systems).

Learning Outcome

The key take aways will be

  • Mistakes one can avoid when scaling elasticsearch in production
  • How to handle resiliency when working with petabyte scale systems

Target Audience

Anyone ranging from beginner to experienced (in elasticsearch)

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Subramaniajeeva Kandasamy
    By Subramaniajeeva Kandasamy  ~  1 year ago
    reply Reply

    Thanks for the feedback Venkatraman and Naresh.

    @Venkatraman

    The monitoring part and alerting part I mentioned is not for single elasticsearch cluster, but for around 40+ clusters we have with our own application architecture.

    As for scaling, I was intending to cover how we scaled our application to put atmost use of resources of elasticsearch with resiliency, which provides enterprise wide search and ingestion across our company.

    And I won't be covering any of the ops side of elasticsearch in this talk.

    @Naresh

    I can skip the Monitoring and Alerting part and focus more on resiliency of our application architecture(which is based on akka streams) and the tuning (with numbers we have).

    But to deliver the scaling part, I will have to give a brief overview of elasticsearch and its tuning(Just adding nodes to cluster not necessarily help scaling without tuning)

    I don't have any blog written myself. But the usecase I built was mentioned in highscalability.com 2 years back.

    Here is the link: http://highscalability.com/blog/2016/8/15/how-paypal-scaled-to-billions-of-transactions-daily-using-ju.html

    • Naresh Jain
      By Naresh Jain  ~  1 year ago
      reply Reply

      Thanks for sharing the details, Subramaniajeeva. Can you please update your outline?

      Also, for the program committee to understand your presentation skills, can you please share a link to past video presentation? If you don't have one, can you please record 1 min trailer of your talk and share that video?

      • Subramaniajeeva Kandasamy
        By Subramaniajeeva Kandasamy  ~  1 year ago
        reply Reply

        Sure. will take it right away.

        • Subramaniajeeva Kandasamy
          By Subramaniajeeva Kandasamy  ~  1 year ago
          reply Reply

          Here is the link for the one min video:

          https://drive.google.com/open?id=1C6RQDAiqi0FGUH3rA3_0Y9Z48xnn-D32

          • Naresh Jain
            By Naresh Jain  ~  1 year ago
            reply Reply

            Hi Subramaniajeeva,

            Thanks for sharing the video. Would it be possible for you to compressed this into a 20 mins talk in which you take a specific use-case and explain how you've scaled Elasticsearch in production (Tuning) and also how you've designed it for resilience?

            • Subramaniajeeva Kandasamy
              By Subramaniajeeva Kandasamy  ~  1 year ago
              reply Reply

              Sure Naresh. Can do that. Let me know when you need updated version of the abstract.

              I'll share about the below during the session:

              • Scaling Elasticsearch in production (Tuning of application)
              • Resilient by design (Application architecture and reactive systems)

              Also, can you please let me know when(date) the session will be happening?

               

              • Naresh Jain
                By Naresh Jain  ~  1 year ago
                reply Reply

                Thanks. Please update the proposal as soon as you can. We are finalizing the sessions now. Your talk will mostly be on Sep 1st.

  • Venkatraman J
    By Venkatraman J  ~  1 year ago
    reply Reply

    Hi SubmramaniaJeeva,

    Thanks for writing the proposal. Could you please change it to focus on the intelligence part of Elasticsearch  or specific topic like @naresh mentioned, rather than talking about Ops work for elasticsearch. Monitoring,alerting and scaling up Elastic nodes is more on the Ops side of DevOps according to me. This conference is mainly for Data science professionals.

     

  • Naresh Jain
    By Naresh Jain  ~  1 year ago
    reply Reply

    Subramaniajeeva, thanks for the proposal. This is certainly a very important topic for many organizations like you pointed out in the abstract. 

    Given that you only have 45 mins for this presentation, I feel you are trying to cover way too much. I fear this might end up being a touch and go (overview) session. If I google, I can find many videos covering high-level topics.

    I would recommend, you to focus on one specific area and go into the depth of it. Share your first-hand insights, which won't be easily available on Google. For example, you could focus just on the resiliency at scale part and strip out all the other stuff. I'm sure there must be something unique that you guys did, which would be worth sharing.

    Also, for the program committee to get more confidence in your expertise, we would like more details. Link to past video presentation or links to articles you've written on this topic.

    Thank you!