Machine learning models are rapidly conquering uncharted grounds with new solutions by proving themselves to be better than the existing manual or software solutions. This has also given rise to a demand for Explainable Artificial Intelligence (XAI) that can be used by a human to understand the decisions made by the machine learning model. The need for XAI may stem from legal or social reasons, or from the desire to improve the acceptance and adoption of the machine learning model. The extent of explainability desired may vary with the aforementioned reasons and the application domain such as finance, defense, legal, and medical. XAI is achieved by choosing machine learning technique such as decision trees that lends well to explainability but compromise accuracy, or by putting additional efforts to develop a secondary machine model to explain the decisions of the primary model. Essentially this leads to a choice between the desired levels of explainability, accuracy, and development cost. In this talk, we present current thinking, challenges and a framework that can be used to analyze and communicate on the choices related to XAI, and make the decisions that can be used to provide the best XAI solution for the problem in hand.