One of the interview questions we sometimes ask of prospective candidates here at Galen is, “How would you find a needle in a haystack?” Over the years we have received answers all across the board, some good, some bad — one of my favorites being to use a can of gasoline in combination with a match. We ask that question to lighten the mood and to see how candidates think on their feet, but also because it illustrates the nature of much of the work we do. Our Data Migration and Archival Implementation teams are quite often looking for a needle in a haystack when building complex and all-encompassing ETL processes to capture various data points from legacy systems.
That work is essential for keeping a patient’s legal medical record intact, but it also reveals hard-to-access data points which can more accurately be used to depict patient population holistically while driving decisions at the point of care. Relevant data points can be found in a variety of formats (structured, unstructured) strewn across multiple systems (EHR, Internet of Medical Things – IoMT — devices, and various 3rd party tools). It’s hard enough to find a needle in a haystack, let alone when that needle is broken into various pieces inside multiple haystacks.
One effective approach to this problem is to build a solution on the Elastic Stack (ELK), a search-focused JSON document-store database. Elasticsearch (ES) is not a one-size fits all product, but it does provide a solid foundation for flexible document storage, full-text search, and powerful analytics, all with fast response times to meet and exceed the demands of modern applications and business needs.
Many organizations, including UCLA, CTcue, and Forcura, have devised valuable products and custom-built solutions by developing on the ELK stack. Galen has extensive experience with these solutions, powered by Elasticsearch, and how they can uncover vital data points through our open-source tools and custom-built solutions.
What might a unified, user-positive experience look like when built on Elasticsearch? In this blog series we will break that question down into three components and evaluate the technical requirements to pull everything together. Each of the individual steps can be found within its own future post:
- Data Collection and Aggregation
- Structure and Storage
- Valuable Data