Customer Reviews flow in across multiple sources in thousands per day. These reviews could be in the form of a transcribed telephone conversation, an online review or, a survey. We want to extract cases of Product Safety from this huge amount of data. Product Safety issues cover less than 1% of the total reviews which may be as high as 70K per day coming from a singular data source. Manual review of these cases to identify potential product-safety threats poses the risk of missing out on such rare events, which may later back-fire on the industry jeopardizing its reputation and liability. Preventive measures, under such circumstances, are extremely crucial and time-sensitive.

Unlike simple cases of imbalanced binary classification, the challenge here is to gather training data. Available training data consists of very few Reportable cases, pertaining to Product Safety, and no Non-Reportable case. This is an extreme case of PU Classification where the class, for which instances are available, is highly imbalanced. However, every PU algorithm would require a lot of instances of the Positive Class, in our case, Reportables. In real life, instances of such rare events are difficult to come by and hence, obtaining enough training data to run a PU Classification is a luxury. In iCASSTLE, we propose a two-staged classification where Stage I leverages three unique components of text mining to capture a representative training data containing instances of both classes in the right proportions, and Stage II leverages results from Stage I to be fed into a semi-supervised set-up. The final Classification is given by Stage II and a top percentage of Stage I results. Furthermore, Reportables are arranged in order of a metric indicating the degree of Reportability. We applied this to multiple datasets differing in nature of Reportables as well as nature of imbalance. iCASSTLE was able to ensure recall maximisation of all such diverse datasets and therein lies its applicability.