Back to projects

Embeddings · Healthcare Apr 2025 to Mar 2026

Medical Document Similarity & EMR System

Abstract.Similarity analysis of electronic medical records (EMR) for a corporate R&D lab.

Role
AI Engineer
Period
Apr 2025 to Mar 2026
Context
Electronics manufacturer, R&D lab (via Human Resocia)
Team
About 20
Table 1Results.
4 Embedding APIs benchmarked
AUROC · F1 Offline evaluation harness for model choice
JSON · CSV EMR editing and export for doctors

01From problem to approach

The problem

Related clinical records are written in very different ways, so keyword search misses them. The records are also confidential.

The approach

Combine classical retrieval with Transformer embeddings, and choose the embedding model by measured accuracy rather than reputation.

02How it works

  1. Input Ingest EMR (JSON) Load electronic medical records and clinical evaluation data.
  2. Step 02 Preprocess Normalize medical text; hybrid rules + ML absorb expression variance.
  3. Step 03 Embed Embeddings from Titan, Azure OpenAI, OpenAI or Cohere.
  4. Step 04 Similarity & ranking TF-IDF, cosine similarity and KNN retrieve and rank related records.
  5. Step 05 Benchmark Offline AUROC / F1 and error analysis pick the best model.
  6. Output Review in Streamlit EMR management UI plus an interactive similarity graph (Pyvis).
Fig. 1Medical Document Similarity & EMR System, step by step.

03What I built

  1. Built similarity search over confidential clinical records: KPI extraction plus TF-IDF, cosine similarity and KNN over Transformer embeddings.
  2. Added hybrid rule + ML scoring to absorb variation in how medical findings are expressed.
  3. Benchmarked embedding models on AWS Bedrock (Titan), Azure OpenAI, OpenAI and Cohere with an offline AUROC / F1 harness, choosing the best model and improving retrieval accuracy.
  4. Developed a Streamlit EMR data-management UI (JSON editing, CSV export) and Pyvis graph views of similarity results for error analysis, improving usability for doctors.

04Technologies

  • Python
  • scikit-learn
  • Transformers
  • Pandas
  • NumPy
  • Amazon Bedrock
  • Azure OpenAI
  • OpenAI API
  • Cohere API
  • Streamlit
  • Pyvis

On request

Request my resume

I share my resume on request. Tell me a little about yourself, and I'll email it to you once I've reviewed your request.

I am a…

Which document?

0 / 100 words

Your details are used only to handle this request and are never shared.