Building an Augmented IT Recruiter
This is an experience report giving challenges while building automated , AI empowered application that converts the mundane job of an IT recruiter to an augmented job.
Problem: Recruiters hated transcribing resume data into long forms to enter into our ATS (Applicant Tracking System) . Using standard resume parsers didn't help because these parsers could not understand most of the technical words in the resumes. We tried buiding our own parser using the GATE library. It worked but GATE is single threaded and runs into memory issues at scale. Also training the parser to recognize thousands of technical words would need us to invest in a large team of professionals. We had to make this work at web scale so that many users could train the parser to recognize technical terms that were common to each user's context.
The UX has to be as smooth as a prosthetic extension. The user experience should get the user dependent on and addicted to using the application. Performance assumes central role - latency of a few milliseconds can send users off to other distractions. Building user retention and loyalty is even more challenging. This report shows how the principles of "Game Thinking" can be used to engage and motivate the users to become trainers of ML/NLP algorithms.
Solution: We built a highly scalable containerized microservice to parse resumes using GATE. We deployed these Docker containers on AWS and used Kubernetes to do auto scaling which led to an amazingly fast resume parsing service which could parse a hundred resumes in less than a minute. We worked around the memory leak issue by implementing the circuit breaker pattern.
According to Raph Koster users derive pleasure by learning to use your application. We built the application to "teach" the users to "train" the parser to recognize terms that are relevant to their specific context. This is supposed to give the users a pleasurable experience and a feeling of empowerment.We will present how the progression of hypothesis-driven sprints is helping us get closer to the goal.
Outline/Structure of the Case Study
This is a 45-minute talk divided into 4
1) Introduction- What is an augmented job- 5 minutes
2) Game thinking principles - 5 minutes
3) Cloud native, scalable, containerized microservice - 15 minutes
4) Training the NLP/ML algorithms - 15 Minutes
4) Q & A - 5 Minutes
Would it make sense to reduce the time talking about the solution and actually demonstrating it? Would it make sense to do a code walk through? I suspect that both these activities need more time and would break the flow of the presentation. Also code walkthrough may not be well understood by everyone. Instead I can provide links to articles to those who want to do a deep dive. Here are 2 such links
Developers will leave the talk with a better understanding of design of microservices. Data Scientists will wrap their heads around leveraging engaging UX to get users to build training sets- an idea that can be used to build any training set.
This is targeted for Developers, Lean UX designers and Data Scientists. Executives and entrepreneurs will find it interesting too.
Prerequisites for Attendees
The participant should be familiar with basic concepts of AWS, Docker, K8s, microservices , machine learning and natural language processing.