This article examines ArnoX, a Flutter accessibility app, and the IEEE research paper behind it: “Multimodal Narration System for Enhancing Accessibility for the Visually Impaired” by Swagat Kumar Samantaray, Mansi Kaushik, Vishal Kumar Ojha, Rishav Mishra, and Abhishek Kumar, published at the 2025 International Conference on Cognitive, Green and Ubiquitous Computing (IC-CGU). The full paper is on IEEE Xplore: https://ieeexplore.ieee.org/document/11337992.

The goal is not to replace the paper, but to explain its engineering context: why OCR alone is not enough, how image and video narration differ, what the research architecture proposes, what the repository actually implements, and where the two diverge.

The information hidden in everyday visuals

Consider a reader who cannot see the packaging of a new medication. The box may contain the drug name, dosage, expiry date, and warnings. A screen reader can read that text if it already exists in an accessible digital format. It cannot read a photograph of the box unless something first extracts the text from the image.

Now consider a photo of a room or a street scene. There may be little text, but the visual content still matters: what objects are present, how they relate, whether a path is clear, or which way a sign points. OCR has nothing to extract from a scene with no readable text. A description is needed instead.

Video adds a third layer. A single frame might show a person, a door, and a chair. Across time, the same sequence might show someone walking toward the door and leaving. Understanding that change requires temporal information, not just one still image.

These are different accessibility problems that need different kinds of machine understanding. That is the central idea behind multimodal narration: combine text recognition, visual description, and speech so the user receives more than one narrow view of the same input.

The paper frames the problem similarly. It notes that visual information is fundamental to daily interaction, while visually impaired people face barriers in accessing digital and printed media. It argues that Braille, screen readers, and mobility aids often lack real-time adaptability, multimodal interaction, and contextual understanding. The paper cites a 2011 World Health Organization figure of 285 million people with visual impairment, including 39 million blind and 246 million with mild or severe impairment. That is a historical reference point; current WHO estimates use a broader definition and a much larger number, so the statistic should be read in context.

The precise number matters less than the dependency: visual information is part of daily life, and existing interaction patterns leave gaps.

The problem and the research motivation

The paper’s problem statement is straightforward: assistive tools tend to operate in isolation. OCR can extract text, TTS can read text aloud, and image captioning can describe a scene, but a user often has to move between separate tools or accept a partial result. A screen reader does not necessarily describe an image. A captioning demo does not necessarily read the text printed inside that image. A video description tool may not expose its result through an accessible mobile interface.

The research proposes an integrated pipeline combining:

• Optical Character Recognition (OCR) to extract visible text.
• Image captioning to describe visual content.
• Video captioning to describe visual content over time.
• Text-to-Speech (TTS) to deliver the result as audio.

The paper’s stated objective is a real-time, AI-powered multimodal narration application. Its three-layer design is:

1. Input layer: capture an image through a smartphone camera or upload one.
2. Processing layer: run OCR and image captioning.
3. Output layer: deliver synthesized speech through TTS.

That is the research contribution in broad strokes. The repository tells a more specific and more complicated story.

Why multimodal accessibility matters

It is tempting to treat OCR as “the accessibility feature.” That is too narrow.

OCR converts pixels that look like text into machine-readable characters. It works well for documents, labels, signs, and screenshots. It is weak when an image is blurry, rotated, poorly lit, handwritten, or full of decorative fonts. It also returns text without necessarily preserving layout, so a receipt, table, or multi-column page can become a stream of words that is technically correct but hard to interpret.

Image captioning takes a different path. A captioning model looks at the whole image and generates a natural-language description: “a blue cup on a wooden table,” or “a person standing beside a bicycle.” This is useful when there is little or no text. But captions can miss small text, hallucinate details, or blur the distinction between what is certain and what is guessed.

Video captioning adds time. Meaning often comes from motion, order, and change. Frame-by-frame descriptions can help, but they can also produce repetitive narration if the system does not aggregate or reason over time. The paper treats video as a frame-wise analysis problem with scene-level descriptions from an external API. That is a practical starting point, not a complete theory of video understanding.

TTS is the delivery layer. It turns the combined result into audio so the user can listen instead of reading a screen. Voice quality, speaking rate, language, and the ability to stop or repeat playback all affect whether the output is genuinely usable.

The modalities complement one another:

• OCR answers: What text is here?
• Image captioning answers: What is shown here?
• Video captioning answers: What is happening over time?
• TTS answers: How can this be delivered without requiring sight?

A system that uses one of these can be helpful. A system that combines them covers more of the information a sighted user takes for granted.

Figure 1. A conceptual multimodal pipeline. OCR handles visible text, captioning handles visual content, video adds temporal descriptions, and TTS delivers the result. Redrawn from the paper’s system description.

Introducing the research system

The IEEE paper describes a Flutter front end connected to a backend ensemble. For images, the app sends input to a Flask API, where OpenCV handles preprocessing and text detection, and Tesseract performs OCR. The extracted content is structured as JSON and stored in Firebase. For video, the paper says input is processed through the Google Studio API, which performs frame-by-frame analysis and returns descriptive captions. The resulting text is displayed and narrated through Flutter TTS.

The paper presents this as an integrated, context-aware experience rather than separate demos. The intended user experience is:

1. Open the mobile app.
2. Capture or upload an image or video.
3. Let the system extract text and/or generate a description.
4. Read the result on screen.
5. Hear the result spoken aloud.

The paper’s own figures show the system architecture (Fig. 1), mobile-to-API interaction (Fig. 2), the OCR and captioning pipeline (Fig. 3), and the text extraction workflow between Flutter and Flask (Fig. 4).

Figure 2. High-level architecture described in the paper, redrawn. The research design routes images through a Flask backend and video through an external frame-analysis API before narration.

The paper uses the TextOCR dataset, a scene-text benchmark of natural images annotated with dense text across font styles, sizes, orientations, languages, and backgrounds. It argues the dataset can support both OCR and captioning, although TextOCR is not a conventional captioning benchmark such as MS COCO. For speech, it uses pre-trained Google TTS models with configurable pitch, rate, and language; the methodology describes a CNN encoder, an RNN/LSTM decoder, attention, and beam search for image captioning.

Those are the research claims. The repository is a separate artifact and does not implement all of them.

Technical architecture: paper versus repository

The paper’s architecture is a client-server design. Flutter is the mobile client. The image path goes to a Flask API, which performs OpenCV preprocessing and Tesseract OCR and hosts a deep-learning captioning model. Video goes to an external service. Firebase stores structured output. Flutter TTS handles narration.

The repository is a mobile-first, on-device design. It uses Flutter for the UI, Firebase Authentication for sign-in, Cloud Firestore for history, Google ML Kit for image labeling and text recognition, and flutter_tts for speech. There is no Flask server, no HTTP client for a custom backend, no Tesseract dependency, no TensorFlow dependency, and no video input path in the Dart code.

That difference changes where data is processed, what works offline, what leaves the device, and which parts of the paper’s evaluation apply to the shipped app.

Figure 3. The repository’s Flutter application structure, based on the actual screen and dependency graph. This is not the paper’s Flask-based architecture.

OCR: from preprocessing to extracted text

The paper’s OCR pipeline starts with a camera or gallery image, then applies grayscale conversion, adaptive thresholding, denoising, and resizing. Tesseract extracts the text, which is post-processed and passed to TTS. The paper’s Fig. 4 shows the Flutter-to-Flask request and JSON response.

The repository takes a different route. In lib/Screens/gallery.dart, _recognizeText creates an InputImage and runs ML Kit’s TextRecognizer:

final inputImage = InputImage.fromFilePath(image.path);
final textRecognizer = TextRecognizer(script: TextRecognitionScript.latin);

try {
  final RecognizedText recognizedText =
      await textRecognizer.processImage(inputImage);
  setState(() { _text = recognizedText.text; });
  Get.to(() => ResultView(
    text: _text, image: image, description: description,
  ));
} catch (e) {
  setState(() { _text = 'Error recognizing text.'; });
} finally {
  textRecognizer.close();
}

This is genuine on-device OCR through Google ML Kit. There is no OpenCV preprocessing in the Dart code, no Tesseract call, and no Flask request. The only language configuration is Latin script. The Android Gradle file includes additional ML Kit language packages for Chinese, Devanagari, Japanese, and Korean, but the Dart recognizer is still initialized for Latin, so those packages are not exercised by this path.

The paper’s 90.96% OCR accuracy belongs to a Tesseract pipeline with preprocessing, not the repository’s ML Kit path.

Figure 4. OCR workflow described in the paper, redrawn. The repository replaces Tesseract and Flask with on-device ML Kit text recognition.

Text-to-Speech: turning the result into audio

The paper says OCR text and the generated image caption are concatenated into one string and passed to TTS. It mentions voice pitch, speaking rate, and language settings, plus a stop-token prediction mechanism to detect the end of narration.

The repository uses flutter_tts directly. In lib/Screens/result.dart, the result screen configures English (United States), rate 0.5, and volume 100:

FlutterTts flutterTts = FlutterTts();

_initTts() async {
  await flutterTts.setLanguage("en-US");
  await flutterTts.setSpeechRate(0.5);
  await flutterTts.setVolume(100);
}

void _speak(String text) async {
  await flutterTts.speak(text);
}

Two differences matter. First, narration is triggered by a Narrate button; it does not speak automatically when the result screen opens. Second, _speak(widget.text) passes only OCR text. The image description is displayed but not spoken. The result screen is multimodal in presentation, but not fully multimodal in narration.

The volume value is also worth flagging. flutter_tts commonly expects a normalized volume between 0.0 and 1.0, depending on the platform implementation. Passing 100 is unusual and should be verified on a physical device. The app does not expose voice selection, pitch control, speed control, pause/resume, or a visible stop control beyond calling flutterTts.stop() in dispose().

The paper describes the Google TTS API, while flutter_tts wraps the platform’s native speech engine (often Google’s on Android, the system synthesizer on iOS). The repository has no Google Cloud TTS client or credentials.

Figure 5. TTS pipeline described in the paper, redrawn. The repository implements a simpler button-triggered flow and speaks only the OCR text.

Image captioning: a research model versus a labeler

The paper’s image-captioning approach follows an encoder-decoder pattern. A CNN encodes the image into visual features. An RNN, typically with LSTM cells, decodes them into a sentence. Attention focuses on spatially relevant regions, and beam search selects a likely sequence during inference. The paper cites “Show and Tell” as background and includes the caption in the combined narration.

The repository has no CNN-RNN captioning model. Instead, getImageDescription in lib/Screens/gallery.dart uses ML Kit’s ImageLabeler with a confidence threshold of 0.7 and joins up to five labels:

final options = ImageLabelerOptions(confidenceThreshold: 0.7);
final labeler = ImageLabeler(options: options);

final List<ImageLabel> labels = await labeler.processImage(inputImage);
final topLabels = labels.take(5).toList();
String description = "The image contains ";
for (int i = 0; i < topLabels.length; i++) {
  description += topLabels[i].label;
  if (i < topLabels.length - 1) description += ", ";
}
return description;

This produces output such as “The image contains table, cup, text, indoor, wood.” It is a label list, not a grammatical caption. It does not describe relationships, actions, or spatial layout. It also inherits the labeler’s vocabulary and confidence behavior. A user may learn that a cup and a table are present, but not that the cup is on the table or that a person is reaching for it.

The README says “object detection” and “image captioning,” but the code uses image labeling; the paper describes a trained attention-based captioning model. Labels, object detection, and generative captioning are different levels of visual understanding and should not be conflated.

Figure 6. Image-captioning pipeline described in the paper, redrawn. The repository substitutes ML Kit image labels for the CNN-RNN captioning model.

Video captioning: temporal understanding, not a still image

The paper’s video path is described as frame-wise analysis through the Google Studio API. Each processed frame yields a textual description, which can be shown or passed to TTS. The paper’s Fig. 3 presents OCR and image/video captioning as a shared workflow, with the server returning descriptive text to the app.

The paper names the service inconsistently: Google Studio API in the overview, Google Cloud Vision API in the Fig. 2 discussion, and Google Studio Server in Fig. 3. Its future-work section mentions Google AI Studio’s Gemini Vision API, suggesting the video path was still evolving.

The repository does not implement video: no picker, player, frame extraction, upload path, or captioning service in pubspec.yaml or lib/. The README lists “video description features” as a future direction, consistent with the code.

Figure 7. Video-captioning workflow described in the paper, redrawn. This path is not present in the current repository.

How the modalities connect

The paper’s integration story is simple: OCR text and image caption are concatenated into a single string, and TTS narrates the combined content. That is a pragmatic fusion strategy. It does not require a learned multimodal model; it treats text and caption as two textual channels and lets speech deliver both.

The repository’s integration is weaker. The result screen displays the image, the label-based description, and the OCR text. The “Narrate” button speaks only the OCR text. The user therefore has to read the description visually or rely on a screen reader to reach it. For an accessibility app, that is a significant gap: the visual description is present in the interface but not in the primary audio path.

A more complete fusion needs an ordering policy, deduplication, and uncertainty handling—for example, “Text reads: …” then “Scene description: …” and “Some details may be approximate.” The paper describes concatenation but no conflict policy; the repository does not concatenate in speech.

Figure 8. End-to-end interaction model. The solid path reflects the repository’s implemented image workflow; the video branch reflects the paper’s proposed path.

The actual end-to-end user workflow

The current flow is visible in the screen files:

1. lib/main.dart initializes Firebase and starts the app.
2. A signed-in user opens Gallery; otherwise the app opens Intro.
3. Intro leads to login or sign-up, both using Firebase email/password authentication.
4. Gallery offers Take a Picture and Gallery, both using image_picker.
5. After selection, getImageDescription runs ML Kit image labeling, then _recognizeText runs ML Kit OCR.
6. ResultView shows the image, label-based description, and OCR text.
7. Narrate calls flutter_tts.speak with the OCR text.
8. storeToFirebase() runs inside build(), writing OCR text and a timestamp to the user’s Firestore history.
9. The drawer opens history, which streams saved documents and lets the user reopen a result in a second result screen.

This is a real, navigable workflow, but narrower than the paper. There is no visible preprocessing stage, no video branch, no automatic narration, and no combined OCR-plus-description speech.

An illustrative walkthrough: a user photographs a printed notice. The labeler recognizes “paper,” “text,” and “indoor,” while OCR extracts the words. The result screen shows the photo, labels, and text; Narrate speaks only the extracted text. The description stays visual, which shows both the value and the current limit.

Flutter implementation details

lib/main.dart initializes Firebase, wraps the app in ScreenUtilInit, and uses GetMaterialApp for navigation:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );
  runApp(const ITSpeech());
}

DefaultFirebaseOptions points to firebase_options.dart, which is gitignored. A fresh clone must generate it with the FlutterFire CLI and add google-services.json on Android; the repository is not a one-command runnable demo without setup.

The UI uses conventional Material widgets (Scaffold, AppBar, Drawer, TextButton, TextFormField), flutter_screenutil for sizing, and GetX mainly for navigation, not reactive state. Screens use StatefulWidget and setState.

There is no service layer: ML Kit calls live in gallery.dart, TTS in result.dart and resultView.dart, and Firestore writes in result.dart. The prototype is easy to follow but tightly couples UI, plugins, and storage.

Error handling is minimal. _recognizeText catches exceptions and sets an error string, but _pickImage and _clickImage use image! after ImagePicker().pickImage, which throws if the user cancels. Firebase auth flows catch FirebaseAuthException in limited ways. There is no retry, timeout, or recovery path for ML failures.

The Android manifest declares READ_EXTERNAL_STORAGE but not CAMERA; image_picker usually launches the system camera through an intent. The iOS Info.plist lacks NSCameraUsageDescription and NSPhotoLibraryUsageDescription, so camera and photo access will fail until those keys are added. The main Android manifest also lacks INTERNET; debug and profile have it, but a release build using Firebase would need it in main.

Technology stack and design choices

The stack fits a mobile prototype: on-device ML reduces backend cost, Firebase removes server management, and Flutter covers two platforms. The trade-off is ML Kit’s label vocabulary, the platform TTS engine, and Firebase’s network dependency. The paper’s Flask, OpenCV, Tesseract, TensorFlow, and cloud-video stack suits a research pipeline where the team controls preprocessing and model architecture, but adds a server, latency, deployment cost, and data-handling questions.

Data flow and model interaction

The repository’s data flow is local-first for image understanding:

1. The user selects an image from the camera or gallery.
2. ImagePicker returns a file path.
3. InputImage.fromFilePath prepares the image for ML Kit.
4. The labeler returns labels; the text recognizer returns OCR text.
5. Results pass to the result screen as widget properties.
6. flutter_tts synthesizes speech from OCR text.
7. Firestore stores OCR text and timestamp under the signed-in user.

The image itself is not uploaded by the Dart code, a privacy advantage over the paper’s Flask path. The OCR text is stored in Firestore, however, so sensitive text can become cloud data. The repository has no visible privacy notice, retention control, or delete-history action. The paper’s flow sends the image to Flask, runs preprocessing and OCR on the server, returns JSON, and stores the result in Firebase; video goes to an external API. Those boundaries create network dependency and require trust in the backend and cloud provider.

Figure 9. Data flow comparison. Solid lines show the repository’s implemented path; dashed lines show the paper’s design that is not present in the current Dart code.

Experimental evaluation and research findings

The paper reports the following OCR results in Section V:

The TTS module received a mean intelligibility score of 4.14 out of 5 in user testing. The paper says participants cited clarity, fluency, and reduced latency as strengths. It also reports that 82% of participants experienced a marked improvement in interpreting visual content.

The training discussion covers 20 epochs. Training accuracy rises from about 55% to nearly 98%, while validation accuracy climbs from just under 50% to approximately 95%. Training loss starts around 0.9 and drops to under 0.1, with validation loss following a similar trend and minimal variance. The paper interprets the narrow gap as evidence of good generalization.

These are the publication’s numbers. Some cautions:

• The paper does not state the OCR test-set size, train/validation/test split, or exact evaluation protocol.
• It does not report captioning metrics such as BLEU, METEOR, CIDEr, or SPICE, even though captioning is central.
• The user study’s participant count, recruitment, and task design are not reported, so the 82% and 4.14/5 figures cannot be independently reconstructed.
• The paper claims real-time operation but provides no latency table or measured end-to-end response time.
• TextOCR is a scene-text benchmark, not a standard captioning dataset. Using it for both tasks is unusual and is not justified with caption-specific evaluation.

The repository does not reproduce these experiments. There is no evaluation script, test dataset, metrics calculation, or benchmark harness. The only test file is the default Flutter counter smoke test, which does not match the app and would fail. The paper’s results belong to the research pipeline, not the current app.

Accessibility, usability, and design considerations

Accessibility is not just “add TTS.” Output must be understandable, controllable, and timely. The paper aims for multi-sensory feedback—text on screen and speech through the speaker—which supports different preferences and residual vision. A narration system also needs clear structure (is this text, a description, or an error?), control (pause, repeat, stop, speed), announced loading and failure states, predictable latency, privacy, and accessible controls with meaningful labels and adequate touch targets.

The repository has basic Material controls, but no explicit Semantics labels, focus management, haptics, or automatic result announcement. The visual result is not spoken, and storing every result in Firestore from build() can create duplicate entries. The paper does not claim WCAG conformance, and neither should this article; WCAG 2.2 and Flutter’s guidance are useful references, but a checklist is not user validation.

Engineering challenges and trade-offs

OCR accuracy is input-dependent. The paper’s preprocessing targets poor lighting and irregular alignment; the repository relies on ML Kit without custom preprocessing, so glare, rotation, small fonts, and complex layouts remain risks.

Labels are not captions. ML Kit labels do not express relationships. “Cup, table, indoor” is not “a cup on a table beside a window”; quality is bounded by vocabulary and confidence.

Narration is incomplete. The app speaks OCR text but not the image description. The paper’s concatenation approach is a better baseline, but it still needs ordering and uncertainty handling.

Video and networking are costly. Frame-wise video analysis multiplies inference cost, payload, and latency; the repository omits it. The paper’s Flask/cloud design needs connectivity and a running backend, while the repository’s on-device image path is better for privacy but Firebase still needs network.

Platform configuration matters. Missing iOS usage descriptions, a release manifest without INTERNET, and a gitignored Firebase options file can turn a working prototype into a setup puzzle.

The result screen writes during build. storeToFirebase() runs in build(), so every rebuild can create another document. Moving it to initState, a post-frame callback, or an explicit save action would be safer.

UI and ML are tightly coupled. ML Kit and TTS calls are embedded in widgets. A service layer would make testing and model swaps easier, and the duplicate ResultView/Resultview screens increase maintenance risk.

Limitations and future improvements

The publication’s limitations include processing latency, multilingual adaptability, and real-time interaction. It also notes that noisy or distorted images are a challenge and that diffusion-based enhancement methods such as anisotropic diffusion and unsharp masking could be integrated later. It proposes faster deep-learning models, gesture commands, multilingual support, context-aware natural language understanding, and improved video summarization.

The repository’s limitations are more concrete: no video path, no generative captioning, no combined narration, no automatic narration, no evaluation harness, no explicit accessibility semantics, no privacy controls, minimal error handling, no service-layer separation, and duplicate Firestore writes.

Reasonable improvements, clearly proposals rather than existing features:

1. Fuse the narration. Concatenate OCR text and the image description with clear spoken labels and an uncertainty phrase.
2. Upgrade the description stage. Replace label lists with a grounded captioning model or a vision-language model with a confidence signal.
3. Add video sampling and aggregation. Extract key frames, caption them, deduplicate similar descriptions, and narrate a short summary.
4. Add preprocessing. Try deskewing, contrast enhancement, and region-of-interest cropping before OCR.
5. Improve accessibility semantics. Add Semantics labels, focus order, haptic cues, and spoken loading/error states.
6. Add playback controls. Pause, resume, stop, repeat, and speed control should be first-class.
7. Move side effects out of build(). Save history explicitly or in initState.
8. Add privacy controls. Let users delete history, avoid storing sensitive text by default, and explain where processing happens.
9. Evaluate properly. Report OCR accuracy on a defined test set, captioning metrics on a captioning benchmark, end-to-end latency, and a user study with a described protocol.
10. Test on real devices. Verify iOS permissions, Android release networking, TTS volume behavior, and camera/gallery flows.

None of these improvements are claimed as implemented.

What developers can learn

The project is a useful case study in building an assistive AI application because it exposes the gap between a research pipeline and a mobile product.

Separate the research claim from the runtime. A paper can describe Tesseract, Flask, and a CNN-RNN model while the app uses ML Kit and a labeler. Both can be legitimate, but they are not the same system. Document the difference.

Design for model uncertainty. OCR returns text even when it is wrong. Labelers return labels even when they are weak. An accessibility interface should communicate confidence and give the user a way to verify or repeat.

Treat TTS as an interaction, not an output. Speaking once is not enough. Users need control over speed, repetition, and stopping. Narration should be integrated into the workflow, not hidden behind an extra tap.

Keep privacy visible. On-device processing is a strong default for personal images. If text is stored in the cloud, say so and provide deletion.

Build a service boundary. ML, storage, and speech should be services behind interfaces. That makes it possible to swap ML Kit for Tesseract, or a labeler for a captioning model, without rewriting screens.

Test the boring failure paths. Cancelled pickers, denied permissions, missing Firebase config, offline auth, and empty OCR results are the paths users actually hit.

Explore the original IEEE research

The full paper is the right place to go for the research methodology, system description, dataset discussion, OCR and TTS results, training curves, and the authors’ own limitations and future work. It also contains the original architecture figures and bibliographic details.

Publication: Swagat Kumar Samantaray, Mansi Kaushik, Vishal Kumar Ojha, Rishav Mishra, and Abhishek Kumar, “Multimodal Narration System for Enhancing Accessibility for the Visually Impaired,” 2025 International Conference on Cognitive, Green and Ubiquitous Computing (IC-CGU), Bhubaneswar, India, Oct. 10–11, 2025, pp. 1–6, doi: 10.1109/IC-CGU67042.2025.11337992.

IEEE Xplore: https://ieeexplore.ieee.org/document/11337992

Repository: https://github.com/vishalkumarojha/ArnoX

If you want to explore the complete methodology and research findings, read the original IEEE publication. The repository is best read alongside it: the paper explains the research intent, and the code shows what the current mobile implementation actually does.

Conclusion

The accessibility problem is not just “make text readable.” It is “make visual information available through a modality the user can use.” OCR, captioning, video understanding, and TTS each answer a different part of that problem. Combining them is where the engineering becomes interesting, and where the hard trade-offs appear: accuracy versus latency, on-device privacy versus cloud capability, rich description versus model uncertainty, and research ambition versus what ships.

The IEEE paper proposes a multimodal narration system that connects these pieces into one accessibility workflow. The ArnoX repository implements a focused, on-device slice of that idea: ML Kit OCR, ML Kit image labels, Flutter TTS, Firebase authentication, and per-user history. It is useful, but it is not the full paper, and the differences are worth understanding rather than hiding.

That gap is the most valuable lesson. A research prototype shows what is possible; an application shows what is practical today. The next step is to close the distance: complete the narration, add video carefully, measure what works, and build controls that make the output usable daily.

References and further reading

1. S. K. Samantaray, M. Kaushik, V. K. Ojha, R. Mishra, and A. Kumar, “Multimodal Narration System for Enhancing Accessibility for the Visually Impaired,” in Proc. 2025 Int. Conf. Cognitive, Green and Ubiquitous Computing (IC-CGU), Bhubaneswar, India, Oct. 2025, pp. 1–6, doi: 10.1109/IC-CGU67042.2025.11337992.
2. IEEE Xplore, “Multimodal Narration System for Enhancing Accessibility for the Visually Impaired,” https://ieeexplore.ieee.org/document/11337992.
3. ArnoX repository, https://github.com/vishalkumarojha/ArnoX.
4. Flutter documentation, https://docs.flutter.dev/.
5. Flutter accessibility guidance, https://docs.flutter.dev/ui/accessibility-and-internationalization/accessibility.
6. Google ML Kit, “Text recognition,” https://developers.google.com/ml-kit/vision/text-recognition/v2.
7. Google ML Kit, “Image labeling,” https://developers.google.com/ml-kit/vision/image-labeling.
8. flutter_tts package, https://pub.dev/packages/flutter_tts.
9. image_picker package, https://pub.dev/packages/image_picker.
10. Firebase for Flutter, https://firebase.google.com/docs/flutter/setup.
11. Google Cloud Text-to-Speech, https://cloud.google.com/text-to-speech.
12. Tesseract OCR, https://github.com/tesseract-ocr/tesseract.
13. O. Vinyals et al., “Show and Tell: A Neural Image Caption Generator,” in Proc. IEEE CVPR, 2015, pp. 3156–3164.
14. TextOCR dataset, https://www.kaggle.com/datasets/robikscube/textocr-text-extraction-from-images-dataset.
15. World Health Organization, “Blindness and visual impairment,” https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment.
16. W3C, “Web Content Accessibility Guidelines (WCAG) 2.2,” https://www.w3.org/TR/WCAG22/.

Most personal productivity systems do not fail because people lack tools. They fail because the tools are scattered. A job application lives in a spreadsheet. A password lives in the browser. A study plan lives in a notes app. A project write-up lives in a document. A deadline lives in a calendar. The cost is not just switching apps; it is the mental tax of remembering where each piece of context belongs.

EaseOut is an attempt to collapse that surface area into one mobile app. It is a React Native application built with Expo, backed by Firebase, and augmented with Gemini for lightweight planning. The UI covers authentication, a dashboard, a password vault, job tracking, projects, profiles, a calendar, a study planner, custom workspaces, and AI-assisted goals.

The repository is also a useful case study because it is not a polished product with a hidden backend. Almost all of the application logic is visible in the client. That makes the architecture easy to inspect, but it also makes the security and reliability trade-offs easy to see.

This walkthrough is based on the repository at commit 357508d and describes the code as written. It does not claim that the app is deployed, that every feature is complete, or that the current configuration is production-ready. Where something is partial or risky, I will say so directly.

What EaseOut actually does

EaseOut is organized around a dashboard with a quick-access grid for eight features. The authenticated navigation stack also includes an account screen and a custom-spaces screen. The app has thirteen screen modules in total: three for authentication and ten for the signed-in experience.

The clearest way to understand the project is to separate what is implemented from what is scaffolded.

That table is intentionally conservative. A screen existing in the navigation stack does not mean every button on it works. Several support rows are visual only, the notification toggle in Account is local state, and the account statistics are placeholders.

The app name in app.json is ease-out, the Android package is com.ease.out, and the repository is still named PassX-Easeout. The README is the default Expo Snack template, and the app.json slug is a Snack UUID. Those details suggest the project began as a Snack experiment and was later moved into a bare Android workflow. The components/AssetExample.js file and the assets/snack-icon.png are leftover traces of that history.

The stack at a glance

EaseOut is built on a deliberately mainstream React Native stack.

Firebase is the entire backend. There is no custom API server, no serverless function layer, and no repository-owned database schema migration system. The client talks directly to Firebase Auth, Firestore, Storage, and Crashlytics.

That choice buys a lot of speed. Authentication, real-time listeners, offline-capable Firestore clients, file uploads, and crash reporting all come from managed services. For a solo-built mobile app, it removes an entire backend deployment surface.

It also moves trust boundaries into the client. Every Firestore path, query, and Gemini request is initiated by the app. Firestore security rules become the only server-side authorization layer, and the Gemini key becomes a client-side secret if it is embedded in the bundle. Both topics matter later.

The app enables the React Native New Architecture and Hermes in android/gradle.properties. It tracks the android/ directory because it uses the bare workflow, and there is no ios/ directory even though the package.json includes an iOS script and app.json contains iOS settings. In practice, this is an Android-first native build.

A few dependency details matter. @react-navigation/native-stack, react-native-calendars, react-native-chart-kit, and papaparse use * ranges in package.json; the lockfile pins them today, but future updates can drift. papaparse, expo-notifications, react-native-paper, and expo-status-bar are declared but not imported directly. react-native-svg is a chart peer. And expo-file-system is imported by the password screen but is not declared as a direct dependency; it is resolved transitively. That last point becomes a real runtime concern in the CSV import flow.

Repository and architecture

The repository is small enough to read in one sitting. The important parts are:

App.tsx
index.ts
src/
  components/
    BackHeader.tsx
    BottomNav.tsx
    CustomButton.tsx
    CustomInput.tsx
    dashboard/
      QuickAccessGrid.tsx
      ScreenTimeChart.tsx
  constants/
    colors.ts
  context/
    AuthContext.tsx
    ThemeContext.tsx
  screens/
    WelcomeScreen.tsx
    LoginScreen.tsx
    SignupScreen.tsx
    DashboardScreen.tsx
    AccountScreen.tsx
    features/
      PasswordManagerScreen.tsx
      JobTrackerScreen.tsx
      GoalsScreen.tsx
      StudyPlannerScreen.tsx
      CalendarScreen.tsx
      ProjectsScreen.tsx
      ProfilesScreen.tsx
      CustomSpacesScreen.tsx
  services/
    firebase.ts
    GeminiService.ts
    NotificationService.ts
  types/
    navigation.ts
android/
.github/workflows/ci.yml

App.tsx is the composition root. It wraps the application in a Crashlytics error boundary, then an auth provider, then a theme provider, then the navigation container. The auth provider must sit above the theme provider because the theme preference is stored on the authenticated user's Firestore document.

Figure 1: The client-side architecture. Most application logic lives in React Native; Firebase supplies auth, data, storage, and crash reporting, while Gemini and Notifee are called from the client.

The layering is straightforward:

• Screens own local UI state and call Firestore directly.
• Contexts own cross-screen state such as the current user and theme.
• Services wrap the external integrations: Firebase initialization, Gemini calls, and Notifee notifications.
• Firebase provides the durable state.

There is no repository abstraction around Firestore. A screen typically imports firestore() and constructs a collection path itself. That keeps the code direct, but it also means query shapes and path conventions are spread across screens rather than centralized in a data layer.

The error boundary is class-based and calls crashlytics().recordError(error) plus crashlytics().log() with the component stack. It catches render and lifecycle errors in the React tree. It does not catch errors inside event handlers, promise rejections, or native crashes. That is normal for React error boundaries, but it is worth being precise about what “Crashlytics error boundary” covers.

The app requests notification permission in RootNavigator on mount. Because the navigator is mounted for both signed-out and signed-in states, the permission prompt can appear before the user has logged in. That is a product decision worth revisiting: asking for notifications before the user understands the value of the app tends to reduce opt-in rates.

Authentication and session routing

Authentication uses Firebase Auth with email and password. The AuthProvider subscribes to auth().onAuthStateChanged, stores the Firebase user in React state, and loads a matching Firestore profile document.

Figure 2: Authentication is driven by Firebase's auth state, not by manual navigation after login. The navigator switches stacks when the user changes.

The core of the provider is deliberately small:

useEffect(() => {
  const unsubscribe = auth().onAuthStateChanged(async (firebaseUser) => {
    setUser(firebaseUser);
    if (firebaseUser) {
      await fetchUserProfile(firebaseUser.uid);
    } else {
      setUserProfile(null);
    }
    setLoading(false);
  });
  return unsubscribe;
}, []);

fetchUserProfile attaches a Firestore onSnapshot listener to users/{uid}. When the document exists, it copies the data into userProfile. That profile snapshot is what supplies the display name and photo URL throughout the app.

Sign-up does three things in sequence:

const cred = await auth().createUserWithEmailAndPassword(email.trim(), password);
await cred.user.updateProfile({ displayName: displayName.trim() });
await firestore().collection('users').doc(cred.user.uid).set({
  uid: cred.user.uid,
  email: email.trim(),
  displayName: displayName.trim(),
  createdAt: firestore.FieldValue.serverTimestamp(),
});

The profile write does not include preferences.darkMode. The theme provider reads that field with optional chaining and simply keeps the light theme when it is absent. That works, but it means the first login has no persisted theme preference until the user toggles the switch.

The navigator is conditional on user:

{user == null ? (
  <>
    <Stack.Screen name="Welcome" component={WelcomeScreen} />
    <Stack.Screen name="Login" component={LoginScreen} />
    <Stack.Screen name="Signup" component={SignupScreen} />
  </>
) : (
  <>
    <Stack.Screen name="Dashboard" component={DashboardScreen} />
    {/* feature screens */}
    <Stack.Screen name="Account" component={AccountScreen} />
  </>
)}

This is the standard React Navigation auth-flow pattern: authentication changes the navigator's screen set, and navigation follows automatically. The login and signup screens do not call navigation.replace after success. That is a good design because it keeps one source of truth for session state.

The login screen maps Firebase error codes to friendlier messages and validates that email and password are present. Sign-up adds a confirmation password check and a minimum length of six characters. There is no email verification step, no password reset implementation, and no account deletion flow. The “Forgot Password?” text on the login screen is a TouchableOpacity with no onPress handler, so it is currently decorative.

The profile photo flow is more complete: updateProfilePhoto uploads to profile_photos/{uid}.jpg, gets the download URL, updates the Auth profile and Firestore document, and updates local state. It overwrites the same storage path each time and has no delete action.

One cleanup issue: fetchUserProfile creates a Firestore listener but does not return its unsubscribe function to the caller. The listener is not explicitly cleaned up when auth state changes, so repeated account switches can leave stale listeners attached. Keeping the unsubscribe in a ref, or restructuring the effect, would fix it.

The Firestore data model

EaseOut uses a per-user document tree. Everything is nested under users/{uid}, which simplifies the mental model and makes it easy to reason about ownership.

Figure 3: The top-level user document branches into flat record collections, nested space tasks, date-keyed calendar tasks, and daily AI/screen-time documents.

The concrete paths are:

Two conventions stand out. First, calendar tasks are keyed by date at the document level, so a query for one day is a direct subcollection read. Second, AI plans and screen-time records use YYYY-MM-DD document IDs. This makes daily caching and lookup trivial.

The trade-off is that all reads and writes are client-driven. There is no backend validation layer for field shapes, allowed status values, or ownership. TypeScript describes the intended shape, but Firestore stores whatever the client sends. A buggy or malicious client can write arbitrary fields to its own subtree if the rules allow it.

The repository does not contain Firestore security rules, which means the access-control story cannot be verified from source. It also has no index configuration. The current queries are simple — orderBy('createdAt', 'desc'), equality filters, and an in query on dates — but a growing app will need composite indexes.

The date-key convention has a timezone trap. Most date keys are produced with:

new Date().toISOString().split('T')[0]

toISOString() returns a UTC timestamp. Around local midnight, the UTC date can be the previous or next calendar day. A user in India at 1:00 AM local time may write a task under the previous UTC date, while the calendar UI is showing the new local date. The calendar screen uses the selected day's dateString from react-native-calendars, which is local-calendar oriented, but “today” defaults and screen-time writes use UTC. A consistent local-date helper would remove a class of off-by-one bugs.

How a feature writes and reads data

The job tracker is a good example of the app's data flow because it is simple and complete. The screen attaches a real-time listener to the jobs collection, sorted by creation time:

const unsub = firestore()
  .collection('users').doc(user.uid).collection('jobs')
  .orderBy('createdAt', 'desc')
  .onSnapshot((snap) => {
    setJobs(snap.docs.map((d) => ({ id: d.id, ...d.data() } as JobApplication)));
    setLoading(false);
  }, () => setLoading(false));
return unsub;

Creating a job writes a new document and immediately closes the modal:

await firestore()
  .collection('users').doc(user!.uid).collection('jobs')
  .add({
    companyName: newCompany,
    role: newRole,
    link: newLink,
    deadline: newDeadline || 'No Deadline',
    status: JobStatus.APPLIED,
    notes: newNotes,
    createdAt: firestore.FieldValue.serverTimestamp(),
  });

The list updates because the onSnapshot listener fires after the write. There is no optimistic local insert. That means the UI waits for the server round trip and the listener callback, which is usually fast but can feel slower on poor networks. The benefit is that the screen never drifts from Firestore's state.

Status updates are equally direct:

await firestore()
  .collection('users').doc(user!.uid).collection('jobs')
  .doc(id).update({ status });

The job statuses are a TypeScript enum: Applied, No Response, Interview, Rejected, and Offer. The UI renders status chips and color-codes them. The deadline is free text ("April 1st", "No Deadline"), not a timestamp. That keeps data entry flexible but makes reminders and sorting unreliable. The link field is stored but not rendered or opened, and the notes are displayed in the details modal.

The same pattern repeats across projects, profiles, passwords, study topics, calendar tasks, and spaces: a screen owns its path, subscribes with onSnapshot, writes with add or update, and deletes with delete. The consistency makes the code easy to follow, but there is no shared validation or error-handling policy. Some writes show alerts; others catch and ignore. That is the main maintainability cost of skipping a data layer.

Calendar tasks add one more wrinkle because they are nested under a date document:

const tasksRef = () =>
  firestore()
    .collection('users').doc(user!.uid)
    .collection('calendar').doc(selectedDate)
    .collection('tasks');

The screen listens to the selected date's tasks and separately loads every date document to draw dots on the calendar. The second effect depends on tasks, so every task change triggers a fresh read of the entire calendar collection. For a small personal dataset that is acceptable; for a long history, it is wasteful. A single date-range query or a maintained “dates with tasks” summary document would scale better.

The AI daily plan pipeline

The most interesting part of EaseOut is the AI daily plan. It is also the part with the most caveats.

The intended flow is simple: when the dashboard mounts, read today's cached plan from Firestore. If a plan exists, render it. If not, gather a small amount of context from jobs and study topics, ask Gemini for a short action list, cache the result, and send a local notification.

Figure 4: The dashboard checks the daily cache first. On a cache miss, it builds context, calls Gemini, writes the parsed plan back to Firestore, and optionally sends a notification.

The prompt builder is explicit about the output format:

return `You are a productivity assistant for ${ctx.userName}. Today is ${today}.

Here is their current status:

**Job Applications:**
${jobList}

**Today's Scheduled Tasks:**
${calList}

**Study Goals:**
${studyList}

Based on this data, create a concise, actionable plan for today.
Return EXACTLY 4-5 short action items (max 12 words each).
Format each item on its own line, starting with a dash (-).
Focus on what's most urgent. Be specific, not generic.
Do NOT include any other text, headers, or explanation.`;

The API call is a direct REST request:

const response = await fetch(`${GEMINI_API_BASE}?key=${apiKey}`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    contents: [{ parts: [{ text: prompt }] }],
    generationConfig: { temperature: 0.5, maxOutputTokens: 300 },
  }),
});

The service extracts candidates[0].content.parts[0].text, splits it into lines, keeps lines that start with -, strips the dash, and maps the result to DailyPlanItem objects. This is intentionally minimal. It does not use a schema, a response parser, or a retry policy. If Gemini adds a preamble or wraps the list in Markdown, the parser silently drops those lines.

The dashboard caching logic is the real orchestration layer:

const todayKey = new Date().toISOString().split('T')[0];
const planRef = firestore()
  .collection('users').doc(user.uid)
  .collection('aiPlan').doc(todayKey);

planRef.get().then(async (snap) => {
  if (snap.exists && snap.data()?.items?.length > 0) {
    setAiPlan(snap.data()!.items as DailyPlanItem[]);
    return;
  }

  const [jobsSnap, studySnap] = await Promise.all([
    firestore().collection('users').doc(user.uid).collection('jobs')
      .where('status', 'in', ['Applied', 'Interview']).limit(3).get(),
    firestore().collection('users').doc(user.uid).collection('studyTopics').limit(3).get(),
  ]);

  const items = await generateDailyPlan(GEMINI_API_KEY, {
    userName: name,
    jobs,
    calendarTasks: [],
    studyTopics,
  });

  await planRef.set({
    items,
    generatedAt: firestore.FieldValue.serverTimestamp(),
  });
  setAiPlan(items);
});

Four observations matter.

First, calendarTasks is hard-coded to []. The prompt has a section for today's scheduled tasks, but the dashboard never supplies them. That is a missed opportunity rather than a prompt bug.

Second, the catch block is empty:

} catch { /* silently skip if key not set */ }

If the key is missing, the model name is wrong, the network is down, or Gemini returns an error, the user sees no plan and no explanation. Silent failure is convenient during development and confusing in production. At minimum, the UI should distinguish “no plan yet,” “generating,” and “generation failed.”

Third, the model constant is gemini-1.5-flash:

const GEMINI_API_BASE =
  'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent';

Google lists Gemini 1.5 Flash as shut down on September 29, 2025, so this constant needs a supported replacement. The repository's placeholder key, AIzaSyDummyReplaceMe, is not a real key either.

Fourth, the API key is a client-side constant. Even if it is moved into an environment variable at build time, it ends up in the app bundle and can be extracted. For a published app, the Gemini call should go through a backend or serverless function that authenticates the user, applies quotas, and keeps the provider key off the device.

The Goals screen reuses generateDailyPlan with a different context. It sends the user's goal as a single study topic and renders the resulting items as local checkboxes. The results are not persisted to Firestore, so leaving the screen loses the generated plan. The screen also uses the same placeholder key and model constant.

The Study Planner's “Ask AI to Explain This Topic” button does not call Gemini at all. It opens a modal, waits 1.2 seconds, and displays a canned three-point explanation:

setTimeout(() => {
  setExplanationText(
    `Here is a simplified explanation of "${title}":\n\n` +
    `1. Core Concept — Understand the fundamental building blocks.\n` +
    `2. Why it matters — Critical for scalability and performance.\n` +
    `3. Key Takeaway — Master this to build robust, real-world systems.`
  );
}, 1200);

That is a prototype affordance, not an AI feature. It should either be wired to a real model or labeled as a placeholder in the UI.

The AI plan is cached per user per day, which is the right instinct. AI calls are slow and potentially expensive, and a daily plan does not need to be regenerated on every dashboard mount. The cache key is UTC-based, so the same timezone caveat applies. The plan is also stored as a single array field. Toggling one item rewrites the entire array:

const updated = aiPlan.map((item) =>
  item.id === id ? { ...item, isCompleted: !item.isCompleted } : item
);
await firestore()
  .collection('users').doc(user.uid)
  .collection('aiPlan').doc(todayKey)
  .update({ items: updated });

For a four-to-five item list, this is fine. If two devices toggle items at the same time, one write can overwrite the other. An array of item documents or a transaction would avoid that.

The password vault and CSV import

The password vault is the feature with the highest security stakes and the largest gap between intent and implementation.

On the surface, it behaves like a simple credential manager. Entries have a website, username, and password. Passwords are masked by default and can be revealed per entry. The copy button writes the password to the system clipboard. Entries can be added and deleted. The list is a real-time Firestore subscription ordered by creation time.

The storage model is the problem: passwords are stored as plaintext strings in Firestore.

await ref().add({
  website: newWebsite.trim(),
  username: newUsername.trim(),
  password: newPassword,
  createdAt: firestore.FieldValue.serverTimestamp(),
});

There is no encryption, no hashing, no field-level cryptography, and no client-side vault key. Firestore transport encryption protects data in transit, and Firebase encrypts data at rest, but neither protects against an authorized reader, a compromised account, a misconfigured security rule, or an administrator with access to the project. For a real password manager, plaintext storage is a release blocker.

The CSV import flow is also fragile.

Figure 5: The import path reads a file, parses it with a simple splitter, writes every entry in one Firestore batch, and reports success. There is no chunking, schema validation, or encryption step.

The file read uses:

import * as FileSystem from 'expo-file-system';
const content = await FileSystem.readAsStringAsync(asset.uri);

Expo SDK 54 moved the legacy file-system API to expo-file-system/legacy. The default expo-file-system entry point exposes the new object-oriented File and Directory APIs. Importing the default module and calling readAsStringAsync is therefore a runtime risk on SDK 54. The import also is not declared in package.json; it is resolved transitively today. Both should be fixed together.

The parser is a hand-rolled comma split:

const lines = csv.split('\n').map((l) => l.trim()).filter(Boolean);
const headers = lines[0].toLowerCase().split(',');
// ...
const cols = line.split(',');

This breaks on quoted fields, embedded commas, escaped quotes, and password values that contain commas. The project already declares papaparse, which handles these cases, but the dependency is unused. Replacing the parser with Papa Parse would remove a large class of import failures.

The writes are batched but not chunked:

const batch = firestore().batch();
parsed.forEach((entry) => {
  const docRef = ref().doc();
  batch.set(docRef, {
    ...entry,
    createdAt: firestore.FieldValue.serverTimestamp(),
  });
});
await batch.commit();

Firestore batches are limited to 500 writes. A CSV with more than 500 valid entries will fail the entire commit. The app reports a generic “Could not read or parse the CSV file” error, which obscures the real cause. Chunking into batches of 500 or fewer, or using a server-side import, would make the feature robust.

There is also no duplicate detection. Importing the same file twice creates duplicate entries. There is no edit flow for an existing password, only add and delete. The clipboard copy is convenient but leaves the password in the system clipboard, where other apps may be able to read it.

A credible vault would need end-to-end encryption with a user-derived key, a server-side or App Check-protected write path, strict Firestore rules, a migration plan for existing plaintext records, and import validation. Until then, this feature should be treated as a prototype, not a place to store real credentials.

Screen time, calendar tasks, and notifications

The dashboard's “Screen Time Analytics” chart looks like device usage data, but it is not. It measures how long the ScreenTimeChart component stays mounted and writes the elapsed hours to a Firestore document when the component unmounts.

let _sessionStartMs: number | null = null;

useEffect(() => {
  _sessionStartMs = Date.now();

  return () => {
    if (_sessionStartMs && user) {
      const elapsedHours = (Date.now() - _sessionStartMs) / 3600000;
      screenTimeRef().doc(todayKey).get().then((doc) => {
        const existing = doc.data()?.hours || 0;
        screenTimeRef().doc(todayKey).set({
          date: todayKey,
          hours: parseFloat((existing + elapsedHours).toFixed(2)),
        });
      });
    }
  };
}, [user]);

This is an app-session proxy, not Android Digital Wellbeing or iOS Screen Time. It does not measure other apps, unlocked-phone time, or time when the component is unmounted. A force-kill can also skip the cleanup, losing the session. The manual “Edit Today” chips are useful for a demo, but the “Screen Time Analytics” label oversells the data. The module-level _sessionStartMs is also shared across component instances; a ref or dedicated session service would be safer.

Calendar tasks are stored under calendar/{date}/tasks. The screen listens to the selected date and draws dots for dates that have documents. Tasks have a title, a type (job, study, or personal), a free-text time, and a completion flag. The UI is clean and the nested model is intuitive. The main issues are the repeated full-calendar read and the UTC date key described earlier.

Notifications use Notifee. The service creates a high-importance Android channel called easeout_reminders, supports immediate notifications, and includes a trigger-based helper for deadline reminders:

export async function scheduleDeadlineNotification(
  notifId: string,
  title: string,
  body: string,
  deadlineDate: Date,
  hoursBefore = 24,
): Promise<void> {
  await ensureChannel();
  const triggerDate = new Date(
    deadlineDate.getTime() - hoursBefore * 60 * 60 * 1000
  );
  if (triggerDate.getTime() <= Date.now()) return;

  const trigger: TimestampTrigger = {
    type: TriggerType.TIMESTAMP,
    timestamp: triggerDate.getTime(),
  };

  await notifee.createTriggerNotification({ /* ... */ }, trigger);
}

The helper is well-formed, but nothing calls it. The only notification sent by the app is an immediate summary after a new AI plan is generated:

if (items.length > 0) {
  sendImmediateNotification(
    `Your AI plan for today, ${name}!`,
    items.slice(0, 2).map((i) => `• ${i.text}`).join('\n'),
  );
}

The practical consequence is that deadline reminders are not implemented, even though the code suggests they are. Job deadlines are free text, so the included parseDeadlineToDate helper would also need a reliable date source before scheduling could work.

The Account screen's notification switch is local React state initialized to true. It does not request permission, does not persist a preference, and does not enable or disable Notifee. The app requests permission once at navigator mount, and the switch is disconnected from that state.

For Android, exact-time notifications can also require exact-alarm permissions depending on the target SDK and use case. The repository does not include a manifest-level exact-alarm request or a user-facing rationale. That is another reason to treat the notification layer as partial.

CI, tests, and the release gap

The repository includes a GitHub Actions workflow named “EaseOut CI.” It runs on pushes and pull requests to main.

Figure 6: The CI workflow installs dependencies, type-checks, scans for secrets, and audits dependencies. It does not run tests, lint, or build an Android artifact.

The workflow is short and readable:

- run: npm ci --legacy-peer-deps
- run: npx tsc --noEmit
- name: Check for hardcoded secrets
  run: |
    echo "Scanning for potential hardcoded secrets..."
    if grep -rn "AIzaSy\|sk-[a-zA-Z0-9]\|PRIVATE_KEY" \
      --include="*.ts" --include="*.tsx" --include="*.js" src/; then
      echo "WARNING: Potential secrets found in source code!"
      exit 1
    fi
    echo "No secrets found."
- run: npm audit --audit-level=high

The job is named “Lint & Type Check,” but there is no lint step and no ESLint configuration. The type check is real: tsc --noEmit uses the Expo base config with strict: true, which is a meaningful guard against type errors assuming dependencies install cleanly.

The secret scan has a concrete false positive. The placeholder AIzaSyDummyReplaceMe appears in both DashboardScreen.tsx and GoalsScreen.tsx. The grep pattern matches AIzaSy, so the step prints both matches and exits with status 1. The scan is doing its job — it is detecting a hardcoded key-shaped string — but the current code guarantees a red build. The right fix is not to weaken the scan; it is to remove the placeholder from source and load the key through a secure configuration path.

There are no test files in the repository. No Jest configuration, no React Native Testing Library setup, no Detox or Maestro flows, and no test script in package.json. The CI pipeline verifies that the TypeScript compiles and that dependencies do not have known high-severity vulnerabilities. It does not verify that login works, that a job can be created, that a calendar task persists, or that a CSV import handles a quoted comma.

That is the biggest quality gap in the project. For a small prototype, type-checking is a reasonable first gate. For a password vault and a data-heavy productivity app, it is not enough. The highest-value tests would be:

• Auth provider state transitions and sign-out cleanup.
• Firestore path construction for each feature.
• CSV parsing with quoted fields, CRLF, and embedded commas.
• Batch chunking above 500 writes.
• AI response parsing with malformed and well-formed responses.
• Calendar date-key behavior around local midnight.
• A smoke test that navigates through each feature screen with a mocked Firebase layer.

The Android release configuration is also not production-ready. android/app/build.gradle uses the debug signing config for the release build:

buildTypes {
  debug {
    signingConfig signingConfigs.debug
  }
  release {
    // Caution! In production, you need to generate your own keystore file.
    signingConfig signingConfigs.debug
    // ...
  }
}

That is fine for local testing and unacceptable for a store release. There is no EAS configuration, no release workflow, no versioning automation, and no store metadata. The CI workflow does not build an APK or AAB. The repository is a source snapshot, not a release pipeline.

Running the project locally

Because EaseOut uses native Firebase and Notifee modules, Expo Go is not sufficient. You need a development build or a local native build.

git clone https://github.com/vishalkumarojha/PassX-Easeout.git
cd PassX-Easeout
npm ci --legacy-peer-deps
npx expo run:android

You will also need Node.js 20, JDK 17 and Android Studio, an emulator or USB device, a Firebase project with Auth, Firestore, Storage, and Crashlytics enabled, your own google-services.json, and a valid Gemini key with a supported model name.

The ios and web scripts are unlikely to work as-is. There is no ios/ directory, and the app depends on native Firebase and Notifee modules that are not configured for web. The Android path is the supported one.

There is no .env file, no app.config.ts, and no runtime configuration layer. The Gemini key is a source constant; Firebase configuration is expected from google-services.json, which is intentionally ignored. The .gitignore also excludes .env, .env.local, .env.production, keystores, and provisioning profiles.

The Firebase console work is not optional: create the Android app with package com.ease.out, enable email/password auth, create Firestore and Storage, add the Crashlytics plugin, and download the config file. The README does not explain any of this. It is the default Snack README, which tells the reader to open App.js — a file that does not exist here. Replacing it would be one of the highest-leverage documentation improvements.

Security and privacy review

The security posture is the most important part of this codebase to discuss honestly.

Plaintext passwords

The password vault stores credentials as plaintext Firestore fields. Anyone who can read the user's Firestore document can read every password. That includes a compromised account, a misconfigured rule, a project administrator, or a malicious client that can authenticate as the user. This is the single largest risk in the app.

Client-side AI key

The Gemini key is embedded in the client. Environment variables in a React Native build are not secrets; they are build-time strings that can be extracted from the bundle. The code also sends the key as a query parameter (?key=...), which is more likely to appear in logs and proxy traces than an x-goog-api-key header. A backend proxy is the correct boundary.

Missing security rules

The repository does not include Firestore or Storage rules. The per-user path structure is a good foundation, but it only helps if the rules enforce it. The minimum Firestore rule is a user-scoped match on users/{uid} with request.auth.uid == uid, applied recursively to subcollections. Storage needs an equivalent rule for profile_photos/{uid}.jpg. Without those rules, the authorization model depends entirely on the deployed project configuration, which is not visible here.

Client-side search over sensitive data

The dashboard search loads entire collections into memory and filters them in JavaScript. That includes the passwords collection, although the search only matches website and username, not the password value. The query still transfers password documents to the client and holds them in memory. For a vault with many entries, that is both a performance and a privacy problem. Server-side search with indexed, non-secret fields would be safer.

Broad Android storage permissions

The Android manifest requests READ_EXTERNAL_STORAGE, WRITE_EXTERNAL_STORAGE, and SYSTEM_ALERT_WINDOW. The app uses the system document and image pickers, which do not require broad storage access on modern Android, and a productivity app has no clear reason to draw over other apps. For an app that handles passwords and personal profiles, requesting more access than necessary is a privacy and review risk. The manifest should be reduced to the permissions the app actually needs.

The auth and account screens also load remote images from Freepik and pravatar. Those are third-party requests and need a license and privacy review before launch.

Insecure release signing

The release build uses the debug keystore. A production release signed with a debug key is not acceptable for distribution. It also makes it easy to accidentally ship a build that cannot be upgraded cleanly.

Positive signals

The project does several things well:

• Firebase Auth is the source of truth for session state.
• Each user's data is namespaced under their UID.
• Server timestamps are used for creation times.
• The .gitignore excludes Firebase config, environment files, and keystores.
• The CI workflow attempts a secret scan.
• The error boundary reports render errors to Crashlytics.
• The app does not log passwords or AI keys intentionally.

Those are good habits. The gap is that the security-sensitive features need stronger guarantees than good habits.

Trade-offs and lessons

EaseOut shows the trade-offs of a client-heavy Firebase app.

A custom backend would add middleware, API routes, validation, deployment, and monitoring. Firebase avoids all of that and gives the app auth, real-time data, storage, and crash reporting on day one. For a solo developer validating an idea, that is rational.

The cost is that the trust boundary moves into the client. Authorization depends on Firestore rules, secrets risk ending up in the bundle, and data shapes are validated only by TypeScript at compile time. The architecture is fast, but it requires discipline in the areas where prototypes are usually weakest.

A small data layer would help. The screens each build paths and listeners independently; centralizing path builders, date keys, query limits, error normalization, batch chunking, and defaults would make correctness fixes apply everywhere. Tests matter too: the app has many CRUD paths, nested collections, date keys, batch writes, and an external API. Type-checking catches shape errors, not behavior errors.

What I would build next

I would sequence the work by risk, not feature count.

1. Fix the password vault or remove it

Do not advertise the vault as secure until it is encrypted. Build end-to-end encryption with a user-derived key, integrate a dedicated secrets manager, or remove the vault from the production build. Plaintext Firestore storage should not ship.

2. Move Gemini behind a backend

A small serverless function can authenticate the user, enforce quotas, build the prompt, call Gemini with a server-side key, and return only the parsed plan. That removes the key from the client and gives the app a place for rate limiting, logging, and model fallback.

3. Update the model and failure states

Replace gemini-1.5-flash with a supported model. Add explicit loading, empty, error, and retry states. Include calendar tasks in the context. Store the generated plan with its context so the user can understand why an item appeared.

4. Replace the CSV parser and chunk writes

Use Papa Parse, validate rows, deduplicate entries, and split writes into batches of 500 or fewer. Fix the expo-file-system import for SDK 54.

5. Add Firestore and Storage rules

Restrict every user path to the authenticated UID, validate required fields, and prevent clients from writing server-managed fields. Add emulator tests and include the rules in CI.

6. Make CI meaningful

Fix the placeholder-key false positive, add linting or rename the job, add unit tests, and add a release build. The current workflow is a start, not a production quality gate.

7. Wire notifications end to end

Use structured deadlines, schedule reminders with Notifee, persist the preference, handle permission denial and exact-alarm constraints, and cancel reminders when a job is deleted or completed.

8. Replace the README and improve the data layer

Document Firebase setup, google-services.json, the Gemini key strategy, the Android build, and current limitations. Centralize paths, date helpers, query limits, and error handling; add pagination and replace full-collection search with indexed queries.

9. Add tests around risky paths

Start with auth state, Firestore path construction, CSV parsing, batch chunking, AI response parsing, and date-key behavior. Those tests will catch the bugs most likely to hurt users.

Conclusion

EaseOut is an ambitious app in a small package. It combines authentication, a dashboard, a password vault, job tracking, projects, profiles, a calendar, a study planner, custom spaces, local notifications, and two AI-assisted workflows. The code is readable, the Firebase model is easy to follow, and the UI is consistent across many screens.

It is also a clear example of the distance between a working prototype and a production system. The vault stores plaintext credentials. The AI key is embedded in the client and the model name is retired. CSV import uses a fragile parser and one unbounded batch. The release build uses a debug key. CI has no tests and currently fails on its own placeholder secret. None of that is hidden, which is exactly why the repository is useful to study.

The fixes are well understood: move secrets and privileged AI calls behind a backend, encrypt the vault, add rules and tests, make dates local, chunk large writes, and replace placeholders with configuration. Those changes would tighten the boundaries the app already has rather than require a rewrite.

That is the lesson: a serverless mobile app can move quickly, but it cannot skip server-side thinking. Firebase removes the infrastructure, not the trust boundary.

References

• EaseOut repository
• Expo SDK 54 changelog
• Expo FileSystem legacy API
• React Native 0.81 release
• React Native Firebase
• Firebase Auth documentation
• Firestore security rules
• Firestore batched writes and transactions
• Firestore quotas and limits
• Firebase Storage documentation
• Firebase Crashlytics documentation
• React Navigation authentication flow
• React Navigation native stack
• Notifee triggers
• Notifee Android channels
• Notifee repository
• Gemini generateContent
• Gemini model deprecations
• GitHub Actions for Node.js
• TypeScript handbook
• react-native-calendars
• react-native-chart-kit

A note on the example. Northwind and its people (Asha the PM, Ben the engineer, Carla the designer) are invented for illustration. They are not a Notion customer story, and nothing about them is evidence of how Notion works internally.

That ease is the tension this article is about. The more kinds of content a workspace can hold, the harder it is to keep understandable and responsive. Prose, checklists, task boards, and calendars are very different shapes of information, yet one product holds all of them without asking people to manage the machinery underneath. How does it support documents, structured records, nested pages, and several views of the same data at once?

This article works from the outside in: the public content model first, then databases, views, linked views, and templates, and finally navigation, search, and performance. The sources are Notion's help center, its developer documentation, and a handful of posts from Notion's engineers.

What this article will and won't claim

Public documentation explains the object model and the behavior you can see. Notion's engineers have also written about the block model, sharding, client-side caching, and their data lake, and those posts are cited where they apply. They do not describe every production detail, and this article does not fill the gaps with guesses. Three labels keep the line visible:

• Documented: stated in Notion's documentation or engineering posts, with a link and, for engineering posts, a date.
• Analysis: general software-engineering reasoning about why a design might work. It is not a claim about Notion.
• Conceptual: an example or diagram invented for this article to teach an idea.

One more limit: this is a research article, not a benchmark. No timing here comes from our own testing. A section near the end describes a small experiment you can run yourself.

One workspace, many kinds of content

Think about what Northwind puts into its workspace: paragraphs and headings, checklists, screenshots, tables, individual tasks, whole projects, and records such as bug reports. A model that treats a page as one long document handles prose well. It struggles the moment someone asks, "Show me everything due this week that's assigned to Ben." A spreadsheet answers that easily, but it squeezes a design rationale or a meeting discussion into a cell.

A workspace tool has to satisfy several requirements that pull against each other:

• Flexibility: people create different kinds of content without asking permission.
• Structure: records have properties, such as status or owner, that can be filtered and sorted.
• Composability: the same information can appear in more than one place without being copied.
• Discoverability: people can find things as the workspace grows.
• Performance: the app does not have to present everything at once.

Pushing hard on one tends to cost another. The rest of this article follows a handful of concepts that balance them: blocks, pages, databases, views, templates, and loading only what the current interaction needs.

The table below fixes the vocabulary. Its definitions follow Notion's help documentation on databases and data sources and on views.

The diagram below is conceptual. It sketches how different kinds of content land in a workspace. It is not a picture of Notion's internal architecture.

Figure 1 (conceptual). Prose and media become page content. Records become database entries, which open as pages and carry properties.

Understanding the block model

What a block is

A block is a single unit of content: a paragraph, a heading, a to-do item, an image, a toggle. In a 2021 engineering post on Notion's data model, Notion describes blocks as the foundation of the product. Text, images, lists, database rows, and even pages are all blocks in that model.

According to that post, every block has a few core attributes:

• an ID, a randomly generated UUID (version 4);
• properties, a bundle of attributes, of which title is the most common (the text of a paragraph, or the name of a page);
• a type, which decides how the block is displayed and how its properties are interpreted;
• content, an ordered list of IDs for the blocks nested inside it;
• a parent, the ID of the block above it, which the post says is used only for permissions.

Documented: Notion Engineering, May 18, 2021.

Why smaller units help

Treating a page as one blob of text makes every operation coarse: reordering a checklist item or turning a paragraph into a heading becomes an edit to one large document. With blocks, each is an edit to a small record.

The same post gives a neat example. The "Turn into" command changes only a block's type, and the stored properties stay put. A checked to-do that becomes a heading and back again is still checked, because the checked property was merely ignored while the block displayed as a heading. Keeping properties independent of display type makes transformations cheap and preserves what the user meant.

Nesting follows the same logic. A block's content list defines what the post calls the render tree, and block types render their children differently: list-style blocks indent them, a toggle shows them only when expanded, and a page block shows them on its own page. Even indentation is structural. Pressing indent moves a block into the content of the block above it.

A simplified picture

Northwind's project page might look like the snippet below. It is conceptual and simplified. It borrows vocabulary from Notion's API documentation but is not an exact production payload.

{
  "page": "Onboarding Revamp",
  "children": [
    { "type": "heading_2", "text": "Goals" },
    { "type": "paragraph", "text": "Shorten the time to a new user's first success." },
    {
      "type": "to_do",
      "text": "Draft the welcome screen",
      "checked": false,
      "children": [
        { "type": "to_do", "text": "Review the copy", "checked": false }
      ]
    },
    { "type": "child_page", "title": "Kickoff meeting notes" }
  ]
}

In the public API, a page's content is a list of typed block objects, and blocks that can nest report whether they have children. A page is itself a special kind of block, and a sub-page appears in its parent as a child_page reference rather than an inline copy. Documented: Notion developer documentation.

Figure 2 (conceptual). The public content model: a workspace holds pages, pages hold blocks, and blocks can nest.

What the model costs

The same post shows the price of this flexibility. Because blocks nest and permissions flow downward, Notion keeps a downward content pointer and an upward parent pointer, and the two need to mirror each other. Blocks were once allowed to appear in more than one content list, which left it unclear which parent they should inherit permissions from, and an unclear parent is unacceptable in a permission system. Finding ancestors by scanning content lists is also slow, especially on the client.

The cost reappears at scale. In a 2024 post on its data lake, Notion explains that a block's permissions are not stored statically. They are built by walking the ancestor tree up to the workspace, and with hundreds of billions of blocks that computation timed out in their data warehouse at the time.

A caution: an API representation and a few engineering posts describe a conceptual data model, not the production schema. The data-lake post does say every item is a block entity in the backend, stored in Postgres with a consistent structure. We treat that as confirmed and leave column-level details alone.

In short: blocks represent many content types uniformly and make small edits cheap, but ordering, nesting, permissions, and synchronization now live in the relationships between blocks.

Pages, blocks, and databases: related but not identical

A page is both a container for content and a destination you navigate to. In Notion's model it is also a block. The 2021 post lists pages among the things that are blocks, and the API guide calls a page a special kind of block with children of its own.

A database is a structured collection of pages. Notion's help center describes a data source as a set of pages, and each row in a table view opens as its own page. A record can therefore carry two kinds of information at once: properties (typed fields) and a body (free-form content). The API guide offers a useful rule of thumb: properties suit structured information such as a due date, a category, or a relationship, while body content suits looser, free-form writing.

Northwind's task "Ship onboarding flow" shows both halves. Its properties are Status, Owner, and Due date. Its body holds a short design rationale, a link to the mockups, and a sub-page of review notes. Filters and boards work on the properties; people reading the task work in the body. Because the record is a page, it can grow from a one-line task into a small document.

A terminology note. Notion's current help documentation says a database is made up of one or more data sources, and its API now treats databases and data sources as separate objects (query a data source). This article says "database" when the distinction doesn't matter and "data source" when it does.

Figure 3 (conceptual). Each database entry combines structured properties with a page body. The diagram shows what the product exposes, not an internal schema.

Database views: different windows onto the same information

Notion's help center lists seven view layouts: table, board, timeline, calendar, list, gallery, and chart. Forms can also be connected to a database. A new database starts as a table, and you add other views alongside it.

Why would one collection need so many presentations? Because Northwind's people ask different questions of the same tasks. Asha, the PM, wants a board grouped by status to see what is stuck. Ben wants a plain list of what is assigned to him. The team wants a calendar showing what lands before launch. The data is the same; the questions are not.

A view is shaped by a handful of controls: layout, visible properties, filters, sorts, and grouping with optional sub-groups. Filters can combine AND and OR logic in groups nested up to three layers deep. Each view keeps its own settings, and filters and sorts can be saved for everyone or kept private to you. Documented: Notion Help, "Views, filters, sorts & groups".

The most important distinction in this section is that changing a view is not changing data. Filtering to Ben's tasks hides everyone else's tasks from that view and alters nothing. Editing a task's status, on the other hand, edits that task's page. Analysis: since each row is a page, it is natural that every view reads and writes the same underlying pages, which is why a status change on the board also shows up in the list. The documentation describes the behavior of views and pages but does not spell out the mechanism.

Here is the before-and-after for Northwind's tasks:

• Unfiltered view: all of the team's tasks, every property visible.
• Personal view: Owner is Ben, sorted by due date. A short list of what he must do next.
• Planning view: a board grouped by Status and sub-grouped by project. A picture of flow rather than a list of items.

Figure 4 (conceptual view model). One logical collection, several configured views. The diagram does not claim that each view runs its own query or that any particular caching strategy is involved.

Linked databases and contextual dashboards

Imagine Northwind copying its task list into the project page, Ben's dashboard, and the weekly review page. It works for a day. Then a status changes in one place and not in the others, and the copies drift apart. Duplication is a consistency problem.

Notion's answer is to keep one source and show it elsewhere. The help center explains that you can create a database that links to an existing data source, so the same pages appear in a new location, and each location gets its own views, filters, sorts, and groups. Three documented behaviors matter. Views you create or delete in the linked location do not change the views on the original. Edits to the data source's title, properties, or pages are reflected in the original. And a linked data source respects the access level of the original database. Documented: Notion Help, "Data sources & linked databases".

Terminology has shifted: Notion's performance guide still mentions a "linked view of a database" in the slash menu, so older and newer wording coexist.

Here is the Northwind walkthrough:

1. Create a Team Tasks database with Title, Owner, Status, and Due date.
2. On Ben's personal dashboard, add a linked view of Team Tasks.
3. Filter that view to Owner is Ben and sort by Due date.
4. On the project page, add a second linked view of the same tasks, grouped by Status.
5. Edit a task from either place and confirm that the source and the other view agree.

The principle is simple: centralize the information, then tailor how it is presented.

Try it yourself. These steps follow the documentation, not a recorded experiment. Menu names change between releases, and some features depend on your plan, so check the current help pages. One practical catch from the FAQ: if you share a page that contains a linked data source, the recipient also needs access to the original database to see it.

Figure 5 (conceptual). One source database surfaced in two contexts, each with its own view configuration. Edits made through either linked view change the source pages.

Templates and modular workflows

Most databases hold the same kinds of pages over and over: weekly meeting notes, bug reports, design specs. Notion's help center uses exactly those examples to motivate database templates, which define a page structure once and replicate it with one click. A template can pre-fill properties (a bug template might set a priority and a product manager), and it can carry any content, including images, embeds, and sub-pages. Once a page has been created from a template, you can edit it freely. Documented: Notion Help, "Database templates".

A few documented details show how templates are scoped. Templates belong to the database where you created them and are not shared across your workspace. A template can be set to repeat daily, weekly, monthly, or yearly. Nesting a database with its own templates inside a recurring template is possible, with limits: three levels, and not inside a daily-recurring template. The docs also warn against filling in a relation property on a template unless you want every page created from it to point at the same related pages.

Northwind turns its project brief into a template with six sections:

• Goals: what success looks like.
• Owner: who is accountable.
• Milestones: dated checkpoints.
• Risks: what could go wrong.
• Decision log: what was decided, and why.
• Follow-up tasks: a place where next steps land.

Every new project now starts with the same skeleton, and teams tweak what they need. Because content is made of independent blocks, a template is a pre-arranged set of them, and a page created from it is free to diverge.

Templates are about workflow consistency and ease of starting, not scale or performance, and Notion's documentation does not present them as a performance mechanism. They matter here because they show how a flexible system stays usable: by making the common path easy.

Loading and performance: what users experience versus what we can prove

Six techniques that often get mixed up

Large interfaces usually avoid presenting everything at once, and a familiar set of general techniques supports that. They are not synonyms:

• Lazy loading defers fetching or building something until it is needed.
• Pagination (chunked fetching) requests data in manageable pieces.
• Virtualization renders only the list items currently visible.
• Caching keeps results nearby so repeated requests do less work.
• Indexing organizes data so matching records can be found quickly.
• Sharding partitions data or load across separate databases.

Analysis: a product with Notion's requirements might plausibly use several of these. What follows separates what Notion has said from what it hasn't.

What Notion has said

Documented: client caching. The 2021 post describes opening a page by trying local data first: in-memory blocks on the web, and a local store called RecordCache (a least-recently-used cache over SQLite or IndexedDB) on native apps. Only when data is missing does the client call the API's loadPageChunk method, which descends the content tree from a starting block and returns the blocks plus the records needed to render them. The post mentions several caching layers around that call but gives no chunk sizes or rules for requesting more.

A July 2024 post brings SQLite caching to the browser through WebAssembly. Notion reports roughly a 20 percent improvement in page-navigation time, but not in first load: loading the new library blocked first load, so Notion made it asynchronous. Notion also found that some slow devices read from disk slowly, so the app races the local cache against the API. These are Notion's own measurements, and we cannot verify them independently.

Documented: sharding. Notion sharded Postgres at the application level, partitioning by workspace ID, and notes that people usually query within one workspace, so most cross-shard joins are avoided (2021 post). The data-lake post reports 480 logical shards: 32 physical instances with 15 each in 2021, then 96 with 5 each in 2023.

Documented: scale. That post also says block data has doubled every 6 to 12 months, from over 20 billion block rows in early 2021 to over 200 billion by mid-2024. It is Notion's own figure at a specific date, with no independent measurement found. It is also easy to misread: the data lake serves offline workloads that tolerate minutes to hours of latency, not page loads.

Indexing. The 2021 post says block text is indexed in the background for Quick Find after a save, and says no more. The data-lake post lists Elasticsearch among downstream stores and names search as a use case, but does not say it serves Quick Find, so we do not claim that.

What the help center says about database speed

Notion's performance guide is the most direct user-facing evidence. It names three culprits: many pages, many visible properties, and complex sorts or filters on formulas and rollups. Its advice: hide unneeded properties, filter on simple properties, avoid long formula and rollup chains, and avoid stacking many inline databases on busy pages. A single linked database with several views keeps only one view open, so only one database listens for updates. Limits include 250,000 rows and 500 properties per database.

Analysis: this suggests cost grows with rows, visible properties, and live views. The "listening for updates" wording fits the subscription model in the 2021 post, where clients subscribe to changes in records they render. Neither source links the two, so treat that as an informed guess.

Figure 6 (generic interaction model). A common pattern for on-demand loading. It is not a reverse-engineered Notion implementation.

The honest summary: Notion has confirmed local caching, sharding by workspace, and background indexing for search. It has described a chunked page-loading call without giving its rules. For virtualization and for exactly which blocks are fetched while you scroll, the public sources reviewed say nothing, so this article says nothing either.

Search, navigation, and information architecture

Hierarchy. Notion's sidebar guide says pages can nest with no limit, and the sidebar splits into Teamspaces (on paid plans), Shared, and Private sections. A full-page database's views appear as nested items there. An older onboarding guide, mirrored here, also describes a breadcrumb trail showing a page's place in the hierarchy. That is a secondary copy, so treat the wording loosely.

Search. The search help page describes a window opened from the sidebar or a shortcut, with recent pages, exact-phrase matching in quotes, sorting (best matches by default, which favors recent edits and titles over page contents), and filters such as title only, creator, teamspace, parent page, and date. It also lists gaps: comments and discussions are not searched, nor are select-style tag values, and most page and person mentions are skipped.

Filters and database search. A filtered view returns a predictable subset of records. Database search is separate: it matches page titles and property values but not page contents, while workspace search covers content inside database pages.

The trade-off is real. Deep nesting keeps a workspace tidy, but it can also hide information from anyone who doesn't know where to look. Search and filters answer different questions. Search helps when you remember a clue: "the kickoff decision about the welcome screen." A filtered view helps when you want the same predictable answer every day: "all overdue tasks assigned to Ben."

We don't describe the sidebar's or search's internal index architecture, because no public source reviewed does.

A conceptual end-to-end example

Let's run Northwind through one launch, tying each step back to an earlier concept.

1. A project page is created from a template. Asha uses the project-brief template. The page starts with goals, owner, milestones, risks, a decision log, and follow-up tasks. (Templates.)
2. The page holds goals, decisions, and links. Each heading, paragraph, and checklist item is an independent block, and sub-pages nest beneath. (Block model.)
3. Work is tracked in a task database. Every task is an entry in Team Tasks, and every entry is also a page. (Pages and databases.)
4. Each task has properties. Owner, Status, and Due date make filtering, grouping, and calendars possible. (Properties versus body.)
5. A linked view surfaces tasks on the project page. It is grouped by Status so the team sees flow at a glance. (Linked databases.)
6. A personal dashboard shows a filtered set. Ben's linked view shows only his tasks, sorted by due date. (Views and filters.)
7. A meeting note captures decisions and follow-ups. The note lives under the project page, and follow-up actions become new tasks in the same database. (Pages, databases.)
8. People navigate to what they need. Asha browses the hierarchy, Carla searches for a clue, Ben opens his dashboard. Nobody treats the workspace as one giant document. (Navigation and search.)

Figure 7 (conceptual). The Northwind workflow. Every arrow corresponds to a step above; there is no claim about infrastructure.

Notice what the story leaves out: where the data lives, how a board refreshes, what happens when two people edit at once. Notion's engineering posts answer parts of those questions, but the experience works because the model is coherent. Inventing infrastructure to make the example sound sophisticated would add nothing.

Trade-offs and design lessons

The model has costs, and they're worth stating plainly.

• Flexibility versus consistency. People can shape a workspace to their needs, and teams can end up with inconsistent structures. Templates help, but they don't enforce anything.
• Hierarchy versus cross-linking. Nested pages are tidy; useful content can still get buried. Search and linked views compensate.
• Rich records versus simple tables. A database entry that is also a page holds more context, but newcomers have to learn that rows are pages, that properties differ from body content, and that a view is not the data.
• Many views versus cognitive load. Each view is relevant to someone, but a long row of tabs becomes something to maintain, and Notion's docs tie live views to load.
• Power versus performance. Rich, interactive interfaces need careful decisions about retrieval and rendering. Notion's explicit limits and tuning advice suggest the cost is real.

Several lessons follow for engineers designing something similar. They are broader product-engineering principles that Notion's design illustrates, not discoveries unique to it.

1. Choose a coherent content model. One uniform unit, with a few well-defined relationships, scales better than many special cases.
2. Separate data from presentation where users need multiple views of the same thing.
3. Make common workflows easy to start and adapt. Templates do this without limiting what comes next.
4. Design navigation for both hierarchy and search. Neither alone is enough.
5. Measure performance instead of assuming that a clean interface is fast. Notion's own caching write-up shows an optimization that helped navigation and hurt first load until it was adjusted.
6. Show complexity only when it's needed. Properties, filters, and view settings stay out of the way until someone asks for them.
7. Record which conclusions are evidence and which are hypotheses. The hardest part of writing about any product's internals is declining to fill gaps with guesses.

How to explore these ideas yourself

You can observe most of what this article describes in a free test workspace in under an hour. Remember that you will be observing product behavior, not private backend implementation. Plan-dependent features may differ.

1. Create a project page.
2. Add headings, paragraphs, and a to-do list, then nest an item under another.
3. Create a task database with title, status, owner, and due-date properties.
4. Create at least two views with different filters or groupings.
5. Add a linked view to another page, if your plan supports it.
6. Create a project template with the six sections from earlier.
7. Edit an entry from different views and note what changes.
8. Record what is visible, what stays synchronized, and what the interface does while you open pages or scroll.

Screenshots or timings are optional. If you capture timings, record the device, browser, workspace size, and steps, and treat the result as a single observation, not a benchmark.

Two things are worth watching: whether the list view agrees after you change a status on the board, and how a page with several inline databases feels next to one linked database with multiple views, as the performance guide suggests. Treat both as observations, not proof of mechanism.

A simple experiment log keeps the evidence honest:

• Date and product version or context:
• Workspace size:
• Action tested:
• Expected behavior:
• Observed behavior:
• Evidence captured:
• Limitations:

Scale is also an information-design problem

Back at Northwind, nobody in the workspace thinks about blocks, shards, or caches. They see a page, a board, a calendar, and a search box. What makes that possible, according to Notion's public material, is a set of ideas that fit together: content in small, uniform units; pages that double as records; databases that separate data from the views people read it through; linked views that show one source in many contexts; templates that make the common path easy; and navigation that combines hierarchy with search.

Notion's engineers have also shown that the design has real costs, from the permission questions that come with nesting to documented limits on database size. Beyond the points confirmed above, we have said little about storage, fetch behavior, or scale, because public evidence does not support more.

For developers, the lesson is that a scalable workspace is not only a storage problem. It is also a problem of deciding what content means, how pieces of it relate, and how a person finds the right part of it. Which of those decisions would you prioritize first in a workspace application of your own: the content model, the views, or navigation? Share your answer in the comments.

References and further reading

Sources accessed on November 2, 2026. Help-center pages change often, so check the current versions. Locale-specific copies of some help pages (for example, en-gb) were the versions read.

Notion engineering posts (primary)

1. Jake Teton-Landis, The data model behind Notion's flexibility, Notion, May 18, 2021.
2. XZ Tie, Nathan Louie, Thomas Chow, Darin Im, Abhishek Modi, Wendy Jiao, Building and scaling Notion's data lake, Notion, July 1, 2024.
3. Garrett Fidalgo, Herding elephants: Lessons learned from sharding Postgres at Notion, Notion, 2021.
4. Carlo Francisco, How we sped up Notion in the browser with WASM SQLite, Notion, July 10, 2024.

Notion Help Center (primary)

5. Views, filters, sorts & groups, Notion Help.
6. Data sources & linked databases, Notion Help.
7. Database templates, Notion Help.
8. Optimize database load times & performance, Notion Help.
9. Search in your workspace, Notion Help.
10. Navigate with the sidebar, Notion Help.

Notion developer documentation (primary)

11. Working with page content, Notion Developers.
12. Query a data source, Notion Developers.

Secondary

13. Navigate your workspace, an educational mirror of an older Notion onboarding guide (used only for the breadcrumb description).

So I built the opposite: an AI-assisted pipeline that moves a job from raw text to a sent application in deliberate stages, with a human approval gate before anything leaves the system. The model can extract, compare, suggest, and draft. It cannot approve or send.

This article walks through the architecture, the data model, the approval and send safeguards, and the parts I would change next. The code lives in two projects:

• PassX-Easeout: an Expo/React Native Android client.
• jobemailer: a Node.js/Vercel serverless backend.

Together they form a working, offline-tested pipeline. Several external integrations still require real credentials and a live end-to-end run, and I call those out explicitly.

What the system does

The current pipeline has eight stages:

1. Capture a job posting by pasting text or sharing it from another Android app.
2. Extract structured fields with an LLM and show the result with per-field confidence.
3. Let the user review and confirm the job before saving it.
4. Match the job against a resume library using five scored criteria.
5. Generate evidence-based resume suggestions without inventing experience.
6. Draft an outreach email grounded in the job and the selected resume.
7. Require a human to review and approve the exact draft, match, and resume.
8. Send through a server-side Apps Script bridge only after approval is revalidated.

The important design decision is not "use an LLM." It is where the LLM stops.

The model produces candidates. Deterministic code validates them. A human approves specific bytes. The server enforces that approval immediately before the external side effect.

Stack and repository shape

The mobile app uses Expo SDK 54, React Native 0.81.5, React 19.1, and TypeScript 5.9. Firebase handles authentication, Firestore, storage, and Crashlytics. Notifee handles local notifications. The app is Android-only in this repository: android/ is committed and ios/ is not.

The backend is plain ESM JavaScript on Node.js 20 or newer, deployed as Vercel serverless functions. It uses @anthropic-ai/sdk, firebase-admin, googleapis, and pdf-parse. There is no build step.

The two projects meet at a small, typed HTTP contract:

The user-facing authenticated routes use a Firebase ID token. The reminder cron uses a separate CRON_SECRET bearer token. The server derives the user ID from the verified token and never accepts a uid from the request body.

Architecture at a glance

The system has four storage and integration boundaries:

• Firestore is the source of truth for a user's jobs and push tokens.
• Google Sheets stores the resume index and extracted resume text.
• Google Drive stores the original PDFs.
• Apps Script owns Gmail delivery and the existing spreadsheet workflow.

The mobile app never talks to Anthropic, Google Drive, Gmail, or Apps Script directly. It only talks to the Vercel API.

The data model: three states, not one

A job is not a single status. It has three separate state axes:

// PassX-Easeout/src/domain/job.ts

JOB_INTAKE_STATUSES = [
  'unparsed',
  'parsed',
  'needs_review',
  'confirmed',
] as const;

JOB_APPROVAL_STATUSES = [
  'draft',
  'pending_review',
  'approved',
  'rejected',
] as const;

JOB_APPLICATION_STATUSES = [
  'draft',
  'applied',
  'interview',
  'offer',
  'rejected',
  'withdrawn',
  'ghosted',
] as const;

This separation matters:

• intakeStatus says whether the posting has been understood and confirmed.
• approvalStatus says whether a human has approved the exact outreach materials.
• applicationStatus says what happened in the real world after sending.

Collapsing these into one status field makes it possible to confuse "the draft looks ready" with "a human approved it" or "the email was sent." Keeping them separate makes each transition explicit.

Jobs live at:

users/{uid}/jobs/{jobId}

The Firestore document ID is stable for the life of the job. It is deliberately not the Google Sheet row number. Sheet rows move; document IDs should not.

The mobile domain model has no React Native, Firebase, or Firestore imports. Firestore timestamps are converted to ISO-8601 strings at the repository boundary. That keeps the domain serializable and testable without a Firebase project.

The repository also reads defensively. Legacy Job Tracker records share the same Firestore collection but use different field names and values such as "No Deadline". Instead of crashing or silently writing a migration, the boundary coerces unknown values to documented defaults. reads are non-destructive, and the canonical pipeline can coexist with older records.

Stage 1 and 2: capture and extract

The user can paste a posting or share text from another Android app. Android ACTION_SEND text is passed into the Smart Intake screen. URLs are treated as text; the app does not fetch or scrape them.

POST /api/intake accepts:

{
  "rawText": "Senior Backend Engineer at Example Corp...",
  "source": "share_android"
}

The server authenticates first, then validates the body, then loads configuration. An anonymous request cannot discover whether the AI feature is configured. The source vocabulary is a fixed list: paste, share_android, manual, email, linkedin, x, referral, or other. The model does not choose it.

The model returns candidate fields and confidence values. The server then validates the result:

• Confidence values must be finite and between 0 and 1.
• Dates must be real ISO-8601 dates.
• Extracted contact email and job URL must actually appear in the supplied text.
• Unknown fields are stripped.
• Missing fields become null with confidence 0.

The raw posting is not echoed back. The client already has it, and returning it would double the response size.

On the mobile side, the review screen highlights low-confidence fields. The current threshold is 0.6. The user can edit everything before confirmation. Nothing is saved to the canonical job collection until the user confirms.

Stage 3: confirm before saving

The review screen is the handoff from "model output" to "user data." It uses the same deadline parsing and display helpers as the manual job form, so an extracted date and a typed date cannot drift into two different formats.

The user can correct any field, then confirms. Only then is the canonical Job written to Firestore. The raw text remains available for later matching and drafting.

This is the first human checkpoint in the pipeline. It is not the approval gate, but it is the point where the model stops being the authority on what the job is.

Stage 4: matching resumes with a bounded model call

Once a job is confirmed, POST /api/match loads the job, loads the resume library, and asks the model to rank resumes.

The matching contract has five criteria:

Each criterion gets an integer score from 0 to 100, a weight, and an optional rationale. The weights are decided by the server and sum to 1. The model does not get to invent the scoring scheme.

Candidate selection is bounded:

• The library is sent in full up to 15 resumes.
• Resumes without extracted text are excluded.
• If there are more than 15 usable resumes, a deterministic lexical pre-filter keeps the top 15 by overlap with the posting.
• The pre-filter is a budget mechanism, not the final ranking. The model still decides the order.

The server validates every part of the response before writing anything. It rejects:

• scores outside 0 to 100,
• non-integer scores,
• unknown or duplicate resume IDs,
• empty rankings,
• a selected resume that is absent from the rankings,
• a selected resume whose score is not the highest,
• a matchScore that disagrees with the selected ranking,
• malformed JSON or prose around the JSON.

On success, the handler writes exactly three fields to the user's job document:

selectedResumeId
matchScore
matchBreakdown

It does not touch the draft, approval status, application status, or sheet mirror fields. Re-running the match replaces only those three fields.

Stage 5: suggestions that cannot invent experience

POST /api/suggestions takes the job, the selected resume text, and the match breakdown. It returns at most four suggestions.

Each suggestion is evidence-based:

interface ResumeSuggestion {
  id: string;
  priority: 'high' | 'medium' | 'low';
  category: SuggestionCategory;
  actionType: SuggestionActionType;
  title: string;
  suggestion: string;
  jobEvidence: string;
  resumeEvidence: string;
}

The category and action vocabularies are deliberately constrained to working with material the resume already contains. There is no "add a certification" category, because naming one would make inventing credentials a valid request.

The server's normalizer checks the model's output against both the job text and the resume text. A suggestion that cites a skill or achievement not supported by either source is rejected. An empty suggestion list is valid and is persisted as empty, not as null. "Nothing worth changing" is a legitimate result.

Suggestions are inert. Nothing edits the resume automatically.

Stage 6: drafting the outreach email

POST /api/draft generates a subject and message from the job, the selected resume, and the match context.

The draft is not treated as trustworthy because a model wrote it. The server has a single source of truth for the text used by its grounding guards, and the normalizer rejects output that introduces unsupported claims. The client then displays the draft for editing.

The approved content is stored on the job as:

draftSubject
draftMessage

These are the exact strings a human will later approve and the exact strings the send path will later require.

Stage 7: approval is a fact about specific bytes

The approval endpoint is where the human gate becomes real.

POST /api/approval accepts only:

{ "jobId": "abc123", "action": "approve" }

The supported actions are review, approve, reject, and reopen. Everything else in the body is ignored. A caller cannot send approvalStatus, approvedAt, approvedBy, applicationStatus, or another user's uid and have it accepted.

The state machine is explicit:

Approving from draft returns a conflict. Approving a rejected application requires reopening first. Re-approving an already approved job returns success without restamping the approval timestamp.

Before entering review, the server checks that the job is confirmed, has a selected resume, has a match result, and has a draft. Rejection is intentionally exempt from those preconditions. A human must always be able to stop a broken application, even if the draft is missing.

The server writes exactly three approval fields:

approvalStatus
approvedAt
approvedBy

It does not change the application lifecycle, the match, the draft, or the resume.

Approval is revoked when the materials change

A human approved this subject, this message, and this resume. If any of them changes afterward, the previous approval no longer describes what would be sent.

The server therefore treats these fields as approval-invalidating:

draftSubject
draftMessage
selectedResumeId
matchScore
matchBreakdown
resumeSuggestions

Content writes run through a Firestore transaction. If the stored status is approved and a material field changes, the transaction applies a revocation patch. This closes the race where an approval and a content update happen at nearly the same time.

That is the difference between "the UI normally clears the approval" and "the stored approval cannot survive a material change."

Stage 8: sending, with revalidation immediately before the side effect

POST /api/send is the only route that can cause an email to leave the system. It is also the most defensive.

The client sends only a job ID. The server:

1. Authenticates the caller and derives the user ID from the token.
2. Loads the job from users/{uid}/jobs/{jobId}.
3. Rejects non-canonical or missing jobs with the same 404.
4. Requires intakeStatus === 'confirmed'.
5. Requires an approved status, a contact email, a company and role, a selected resume, a match, and a non-empty draft.
6. Resolves the selected resume against the private resume library.
7. Requires the resume to have a server-side Drive file ID.
8. Rejects jobs that already have emailSentOn, gmailThreadId, or applicationStatus === 'applied'.
9. Creates an expiring send claim to serialize concurrent requests.
10. Re-reads the job after claiming it.
11. Compares the approval material with the material recorded before the claim.
12. Rechecks approval immediately before calling the external bridge.
13. Sends only the approved subject and message, plus the resolved Drive file ID.
14. Confirms the result through the Apps Script jobs action and requires a persisted Sent row.
15. Persists applicationStatus, emailSentOn, and gmailThreadId.

The claim is important. Two simultaneous requests must not both reach Gmail. If the Apps Script call times out after the mail was actually sent, a retry is stopped by Apps Script's duplicate confirmation rather than creating a second application.

The send path also resolves the selected resume on the server. The mobile app never receives the Drive file ID. The client cannot swap in a different attachment after approval.

The backend is intentionally boring

The serverless constraints shaped the design:

• Vercel functions have no durable process memory.
• The filesystem is not durable between invocations.
• Cold starts are real.
• Most AI routes have a 30-second ceiling.
• The reminder cron runs once per day at 09:00 UTC with a 60-second ceiling.

So the backend stays stateless. There is no in-process vector index or local embedding model. A module-level cache would be a performance optimization at best, never a correctness dependency.

Every route is built around dependency injection. Handlers export a createHandler(overrides) factory, and tests inject a token verifier, a Firestore double, a Sheet client, a model response, and a clock. That is why the test suite can exercise the full approval and send paths without contacting Firebase, Google, Anthropic, or Gmail.

Configuration fails closed. Missing credentials produce a controlled "not configured" response after authentication, not a stack trace and not a configuration dump. Provider errors are logged server-side and replaced with fixed client messages because model output can quote the job posting, and SDK errors can contain request URLs or account identifiers.

The mobile client is a thin, typed boundary

The mobile app has one HTTP client, src/api/client.ts. It handles:

• base URL resolution,
• bearer token attachment,
• JSON parsing,
• timeout handling,
• cancellation,
• network, timeout, auth, and malformed-response errors.

The API modules validate response envelopes instead of trusting them. A server that returns an older schema version is treated as incompatible. Unknown fields are dropped. This keeps a newer backend from putting unlabelled data in front of an older client.

The Firestore repository is the only place that knows about Firestore. It converts timestamps, validates enums, strips approval fields from content patches, and runs approval revocation inside transactions. The domain layer does not know Firestore exists.

That separation is not architectural decoration. It is what makes it possible to test the pipeline, reason about state transitions, and change the persistence layer without rewriting the UI.

Reminders and Android sharing

The reminder path is separate from the AI pipeline. A Vercel cron calls POST /api/reminders/run with a bearer CRON_SECRET. The route uses Firebase Admin credentials to find deadline and follow-up events, creates deterministic per-user event claims in Firestore to deduplicate repeat runs, and delivers through Expo push. Telegram is optional and requires all three Telegram variables together.

The mobile app also schedules stable local notifications for deadlines and follow-ups and cancels them when a deadline is removed or a job is archived. Reminders only notify. They never send a follow-up email.

Android sharing is intentionally narrow: shared text is passed into Smart Intake for review. URLs are not fetched or scraped.

What I verified, and what I did not

I ran the offline checks while preparing this article:

# Backend
cd jobemailer
npm test

# Mobile
cd PassX-Easeout
npx jest --runInBand
npx tsc --noEmit

Results:

• Backend: 568 tests passed, 0 failed.
• Mobile: 499 tests passed across 24 suites, 0 failed.
• Mobile TypeScript: clean, exit 0.

Those tests are hermetic. They use injected fakes and do not contact Firebase, Google, Anthropic, Gmail, Apps Script, Expo, or Telegram.

What I did not verify for this article:

• A live Firebase sign-in and Firestore read/write.
• A live Anthropic extraction, match, suggestion, or draft.
• A real Google Sheets/Drive resume sync.
• A real Apps Script deployment and Gmail send.
• A production Android build with real Firebase configuration.
• Vercel deployment state and remote environment variables.

So the correct description is: implemented in code, offline-tested, and not live-verified end to end. That distinction belongs in the article, not in a footnote.

Setup outline

The backend needs, at minimum:

APPS_SCRIPT_URL
APPS_SCRIPT_TOKEN
GOOGLE_SERVICE_ACCOUNT_JSON
GOOGLE_RESUME_FOLDER_ID
GOOGLE_RESUME_SHEET_ID
ANTHROPIC_API_KEY
CRON_SECRET

Alternatively, Firebase Admin can use the split credential variables:

FIREBASE_ADMIN_PROJECT_ID
FIREBASE_ADMIN_CLIENT_EMAIL
FIREBASE_ADMIN_PRIVATE_KEY

The Apps Script project needs a DASHBOARD_API_TOKEN script property with the same value as APPS_SCRIPT_TOKEN, and it must be deployed as a web app with access set to "Anyone."

The mobile app needs:

EXPO_PUBLIC_JOBEMAILER_API_URL
EXPO_PUBLIC_EAS_PROJECT_ID

The app also needs a real android/app/google-services.json for package com.ease.out. It cannot run in Expo Go because of the native Firebase modules.

Do not put server secrets in the mobile bundle. EXPO_PUBLIC_* values are public by definition.

What I would change next

1. Per-user resume scoping

The current resume library is a shared Google Sheet. It is effectively single-candidate, and any authenticated caller can read the same resume list. Before this becomes a multi-user product, the library needs real per-user scoping, or the API needs to enforce ownership at the resume level.

2. Retrieval only after measurement

There is a written RAG proposal in the repository, but it is not implemented. That is the right order. The current match path already sends a bounded set of resumes in one model call. Adding embeddings would introduce cold-start latency, index persistence problems, and another failure surface. Retrieval should be driven by measured retrieval quality, not by the fact that RAG is fashionable.

3. Commit Firestore rules

The mobile README documents that Firestore rules are managed in the Firebase console and are not in the repository. That is not a safe production posture. Rules should be versioned, reviewed, and tested like application code.

4. Clean up stale documentation and legacy screens

The codebase moved faster than parts of its documentation. For example, the canonical Job domain file still contains a comment saying the model is not adopted by any screen, while the main pipeline and intake review screens already use it. The legacy Job Tracker screen still reads and writes a different shape in the same collection.

That is tolerable during a migration, but it is dangerous in an article about correctness. The next step is to make the documentation match the code and retire or isolate the legacy path.

5. Finish the release path

The Android release build still needs proper signing. The debug configuration is not a shipping configuration. Real Firebase configuration, an EAS project ID, and a production keystore are operational prerequisites, not afterthoughts.

The lesson: separate generation from authority

The most useful idea in this project is not the prompt. It is the authority model.

The LLM is allowed to:

• propose structured fields,
• rank resumes,
• suggest improvements,
• draft a message.

It is not allowed to:

• decide that a job is confirmed,
• approve an application,
• mark an application as sent,
• choose a different resume after approval,
• write arbitrary fields to Firestore,
• call Gmail directly.

Those decisions belong to deterministic server code and, at the approval gate, to a human.

That separation makes the system slower than a one-click auto-applier. It also makes it something I would actually connect to a real inbox.

If you are building with LLMs in a workflow that has real-world side effects, the useful question is not "How do I make the model more autonomous?" It is "Where is the smallest, most explicit gate I can put between the model's output and the irreversible action?"

For a job application, that gate is a human reading the exact email that will be sent.

Code map

If you want to follow the implementation:

• PassX-Easeout/src/domain/job.ts: canonical job model and state vocabularies.
• PassX-Easeout/src/data/jobs/firestore.ts: Firestore repository, normalization, approval revocation.
• PassX-Easeout/src/api/*.ts: typed HTTP clients and response validation.
• PassX-Easeout/src/screens/JobIntakeScreen.tsx: capture flow.
• PassX-Easeout/src/screens/JobIntakeReviewScreen.tsx: extraction review and confirmation.
• PassX-Easeout/src/screens/JobPipelineScreen.tsx: main pipeline orchestration.
• PassX-Easeout/src/components/jobs/: match, suggestions, draft, approval, and job cards.
• jobemailer/api/*.js: serverless route handlers.
• jobemailer/lib/approval.js: approval transition table and approval material.
• jobemailer/lib/send.js: claim, revalidate, send, confirm, persist.
• jobemailer/lib/claude.js: the single model provider boundary.
• jobemailer/lib/jobs.js: transactional job writes and ownership paths.
• jobemailer/apps-script/Code.gs: Gmail/Sheets bridge.
• jobemailer/test/: backend tests with injected dependencies.

The code is deliberately unglamorous in the places that matter: authentication, validation, state transitions, idempotency, and the final send. That is where an AI workflow earns the right to touch a real inbox.

This article walks through a repository that treats those two problems — probability calibration and decision-threshold selection — as first-class engineering artifacts around a frozen DenseNet121 model. The project does not train a new architecture or claim a new calibration algorithm. Its contribution is a controlled, reproducible study: one fixed checkpoint, one cached prediction table, validation-only parameter fitting, held-out evaluation, patient-level statistical inference, error analysis, Grad-CAM, and a read-only dashboard built on top of the verified evidence.

This is a research and engineering case study. It is not a clinically validated system, and nothing in the repository should be used for diagnosis or treatment decisions.

The problem: ranking is not deciding

The dataset is NIH ChestX-ray14, restricted to two findings: Cardiomegaly and Effusion. The task is multi-label binary classification. The model is a torchvision DenseNet121 with an ImageNet initialization and a two-output linear head. Training uses BCEWithLogitsLoss with positive-class weights derived from the training prevalence ([39.945, 7.552]), because both findings are rare relative to the full cohort.

That weighting is useful for learning from imbalanced data, but it has a side effect: it pushes predicted probabilities upward. A model can rank well while being badly over-confident. The repository measures this directly:

The ranking metrics are respectable. The operating point is not obviously appropriate. The project therefore asks two separate questions:

1. Calibration: are the raw sigmoid probabilities meaningful as probabilities?
2. Thresholds: which per-label operating point should be used, and how does it transfer from validation to test?

A third question — external generalization — is part of the design but remains unanswered because no external cohort is present in the repository.

Who this is for

The project is aimed at machine-learning engineers, applied researchers, and students who want to see how a model-evaluation study is assembled as a reproducible software system. It is not a clinical product, and the dashboard is a demonstration layer, not a medical device.

What the project actually does

The repository implements a complete research pipeline around a frozen control model:

The most important design choice is that the model is not the object of study. The checkpoint is frozen and hash-anchored. What changes between experiments is only how the model's logits are calibrated and thresholded. That makes the comparison a genuine decomposition rather than a sequence of unrelated model runs.

Technology stack and engineering choices

The stack is intentionally conservative. There is no experiment tracker, no database, no web framework, and no CI configuration. The repository uses JSON as the canonical machine-readable result format, CSV as a tidy projection, and PNGs as derived views. That choice makes the evidence easy to inspect, diff, and test without standing up additional infrastructure.

Project structure

Capstone/
├── src/
│   ├── config.py                 single source of paths, labels, splits, settings
│   ├── dataset.py                NIH data loading, transforms, dataloaders
│   ├── model.py                  DenseNet121 encoder + two-label classifier
│   ├── train.py                  frozen baseline training recipe
│   ├── evaluate.py               original evaluation entry point
│   ├── inference.py              cached logits/probabilities per split
│   ├── metrics.py                AUROC, AUPRC, confusion metrics
│   ├── calibration.py            ECE, Brier, NLL, temperature scaling
│   ├── thresholds.py             five decision-threshold policies
│   ├── fit_parameters.py         fit and freeze validation-only parameters
│   ├── experiments.py            experiments 1-4
│   ├── error_analysis.py         strata, high-confidence errors, case list
│   ├── gradcam.py                Grad-CAM overlays and bbox cohort
│   ├── external_eval.py          CheXpert pipeline (blocked on data)
│   ├── statistics/               patient bootstrap, DeLong, permutation tests
│   ├── research_freeze.py        freeze record
│   ├── master_results.py         canonical master table
│   ├── uncertainty.py            bootstrap confidence intervals
│   ├── paper_artifacts.py        tables, figures, exports, summary
│   └── paper_build.py            evidence map and paper package
├── research_demo_app.py          read-only Gradio dashboard + live inference
├── demo_app.py                   older multi-modality demo (legacy)
├── tests/                        unittest suite
├── data/                         raw and processed data (git-ignored)
├── outputs/                      checkpoints, predictions, metrics, figures, paper
└── docs/                         scope, baseline, experiments, reproducibility

The legacy ECHO, MRI, and fusion modules are kept in the repository but are explicitly out of scope for the X-ray research claims. The README labels them LEGACY, and the frozen X-ray baseline does not depend on them. The root-level demo_app.py remains for the older four-modality demonstration; the research dashboard is research_demo_app.py.

System architecture

The architecture separates five concerns:

1. Data and split management. The NIH metadata is joined to images on disk, and the split is frozen to data/processed/split_index.csv. The split is patient-level, not image-level, so images from one patient do not appear in multiple splits.
2. Frozen model and inference. src/inference.py loads the checkpoint, applies the evaluation transform, and writes one cached prediction table per split. The cache key includes the checkpoint SHA-256 prefix, so a different model cannot silently reuse an old model's predictions.
3. Validation-only parameter fitting. Temperature scalers and threshold policies are fitted on validation predictions and written to JSON. The runtime guards refuse to fit on test, external, or chexpert splits.
4. Evaluation and statistics. Experiments read the cached table and the frozen parameters. They never retrain the model. Statistical inference is a separate layer over the same frozen predictions.
5. Evidence and presentation. Results are written as JSON, CSV, and figures. The paper package and the Gradio dashboard are both consumers of those artifacts; neither is allowed to become a second source of truth.

The important boundary is between parameters and results. A temperature or threshold is a parameter: it is fitted on validation and then frozen. An AUROC or F1 on test is a result: it is measured once and never used to tune anything. This is enforced in code, not just documented.

How the application works

There are two workflows worth understanding: the offline research pipeline and the live dashboard inference path.

Offline research workflow

The steps in prose:

1. Freeze the control. The DenseNet121 checkpoint is copied into outputs/checkpoints/baseline/, hashed, and recorded in a manifest. The split index is frozen to CSV. The model is never retrained during the research phases.
2. Evaluate the baseline. src.baseline_eval reads the cached predictions and computes AUROC, AUPRC, sensitivity, specificity, precision, F1, and confusion counts at threshold 0.50. This is the control row for every later comparison.
3. Cache the logits. src.inference stores the raw logits, not only the probabilities. That matters because temperature scaling is defined on logits: sigmoid(logit / T). The cache is keyed by checkpoint hash and split.
4. Fit parameters on validation only. src.fit_parameters fits one temperature per label by minimizing validation NLL. It then fits five threshold policies on validation predictions, once in raw probability space and once in calibrated probability space.
5. Run the experiments. Experiments 1–4 measure calibration quality, threshold policies on raw and calibrated probabilities, and robustness to binning, constraint targets, and validation resampling.
6. Measure uncertainty. The upgraded statistical layer uses a patient-level cluster bootstrap, paired permutation tests with Holm correction for the two primary hypotheses, and DeLong confidence intervals for test AUROC.
7. Analyze failure modes. Error analysis computes confusion strata, high-confidence errors, cross-label overlap, and per-patient error concentration. Grad-CAM then runs on a deterministic case list and on a fixed cohort of test images that have NIH bounding boxes.
8. Build the evidence package. src.master_results, src.uncertainty, src.paper_artifacts, and src.paper_build derive the master table, confidence intervals, figures, tables, evidence map, and claim audit from the frozen artifacts.
9. Present. The Gradio dashboard reads the same artifacts. It does not recompute the research results.

The A/B/C/D decomposition

The central experiment isolates the two knobs:

Two results stand out:

• Arm B equals arm A at threshold 0.50. Temperature scaling is monotone. For a positive temperature, sigmoid(z / T) >= 0.5 is equivalent to sigmoid(z) >= 0.5, so the decisions at 0.50 are identical. The calibrated probabilities change; the fixed-threshold decisions do not.
• Arm D equals arm C in operating-point metrics. The thresholds were refitted in the calibrated space, so the chosen operating points land on the same confusion matrices. The numeric thresholds differ — raw F1-optimal thresholds are 0.9021 and 0.7699, calibrated F1-optimal thresholds are 0.8666 and 0.7035 — but the decisions are monotone-equivalent.

The primary endpoint is the paired difference between arm D and arm A on test F1:

The confidence intervals come from 5,000 patient-level bootstrap resamples with a fixed seed. The bootstrap resamples patients, not individual images, so images from the same patient stay together. The permutation test uses the same paired structure, and Holm correction controls the family-wise error rate across the two declared primary hypotheses.

The practical interpretation is not "calibration improves F1." The result is more specific: choosing a validation-fitted operating point changes the decision metrics; temperature scaling changes the probability scale and the proper scoring rules, but it does not change ranking and does not by itself change decisions at a fixed 0.50 threshold.

Calibration: a deliberately honest result

Temperature scaling is fitted on validation by minimizing negative log-likelihood. The fitted temperatures are:

• Cardiomegaly: 1.1865
• Effusion: 1.3979

On test, the proper scoring rules improve:

NLL and Brier go down. Binned ECE goes up. This is not hidden or rounded away. It is one of the most useful findings in the repository.

Why does ECE rise? Temperature scaling is optimized for NLL, not for ECE. In this cohort, the score distribution is heavily concentrated at low probabilities, and the mean confidence exceeds prevalence in the low-probability region. In that regime, ECE behaves approximately like mean(probability) − prevalence. For Cardiomegaly, the raw mean confidence is about 0.132 and the prevalence is 0.026, giving an ECE of roughly 0.106. After calibration, the mean confidence moves away from prevalence, so the binned ECE increases even though the likelihood improves. The repository reports both the improvement and the regression, and a test asserts that AUROC and AUPRC are exactly invariant to temperature.

This is a good example of why "calibration" should not be treated as a single number. NLL, Brier, and ECE answer different questions, and a monotone transform can improve one while worsening another.

What the errors look like

At the calibrated F1-optimal operating point, the error profile is asymmetric:

The rare label is not just harder; its failures are more often confident failures. Cardiomegaly has far fewer errors in absolute terms, but almost half of them carry a score of at least 0.90. Effusion has four times as many errors, but only 8.5% are high-confidence.

The fixed-0.50 error analysis also shows that failures are correlated across labels and concentrated across patients. At threshold 0.50, 585 images are wrong on both labels, 4,449 are wrong on at least one, and the Jaccard overlap of the two error sets is 0.1315. The 4,449 error images come from 1,317 patients, and the worst patient accounts for 60 error images. That concentration is descriptive, not causal. It is a reason to use patient-level resampling and a reason to look at per-patient failure modes rather than treating every image as independent.

The Grad-CAM analysis adds a qualitative layer. On a deterministic set of 16 TP/FP/FN/TN cases, the heatmaps show where the model's final convolutional block is most active. On a separate fixed cohort of 43 test images that have NIH bounding boxes, the pointing game puts the heatmap argmax inside the box for 21 images (48.8%), and the mean heat-mass concentration ratio is 2.60, with 38 of 43 cases above 1. That is a sanity check that the maps are not random, not evidence of clinically valid localization.

Live inference workflow

The dashboard's live tab uses the frozen checkpoint when it is available. The flow is intentionally simple:

The live path does not fit anything. It loads the frozen temperature values and the frozen calibrated F1-optimal thresholds, applies them, and returns a per-label table. The UI also reports when the checkpoint is unavailable; the artifact tabs continue to work from the stored JSON and CSV files.

The dashboard has tabs for the research overview, live X-ray demonstration, calibration, A/B/C/D decision policy, statistical evidence, threshold stability, error analysis, decision-policy comparison, prevalence sensitivity, and reproducibility/limitations. A "Presentation" mode hides source-path details, while a "Technical" mode exposes them. That split is useful for a faculty demo: the audience sees the evidence, while the presenter can reveal provenance when asked.

Data flow and data model

The dataset has 109,312 usable images from 29,720 patients. The patient-level split is 70/15/15:

The split table's Effusion label sum for test is 1,996, while the cached prediction support is 1,997. One test image is a truncated PNG, and the dataset loader substitutes the next readable sample's label rather than crashing. This is a documented data defect in the frozen control, not a silent repair. The repository keeps the defect to preserve reproducibility of the baseline and records it in the audit documentation.

There is no relational database in this project. The data model is a file-based evidence store:

• Raw prediction table (one CSV per split): Image Index, Patient ID, split, true_<label>, logit_<label>, prob_<label>.
• Frozen parameter files: temperature_scalers.json, thresholds_val.json, thresholds_calibrated_val.json.
• Result artifacts: JSON payloads for experiments, error analysis, statistics, and the research summary.
• Derived artifacts: tidy CSVs for tables and figures, plus PNG plots and Grad-CAM overlays.

Storing logits is a deliberate choice. If the pipeline stored only probabilities, it would be impossible to apply a temperature transform later without re-running the model. Storing logits also makes the monotone-invariance checks direct and testable. The cache filename contains the checkpoint SHA-256 prefix, so the prediction table is tied to the exact model that produced it.

Deep dive: validation-only temperature scaling

The calibration code is a good example of a small module with a strong contract. The key guard is explicit:

# src/calibration.py
FORBIDDEN_FIT_SPLITS = ("test", "external", "chexpert")

split_l = str(split).lower()
if any(bad in split_l for bad in FORBIDDEN_FIT_SPLITS):
    raise ValueError(
        f"refusing to fit calibration on split '{split}': calibration may "
        "only be fitted on validation (or training) predictions"
    )

The fit itself is a one-dimensional bounded optimization over a scalar temperature per label:

# src/calibration.py
if np.unique(y).size < 2 or np.allclose(logits, logits[0]):
    return 1.0

lo, hi = float(bounds[0]), float(bounds[1])
try:
    from scipy.optimize import minimize_scalar

    res = minimize_scalar(
        lambda t: _nll_for_temperature(logits, y, t),
        bounds=(lo, hi), method="bounded",
        options={"xatol": 1e-6},
    )
    if res.success and np.isfinite(res.fun):
        return float(res.x)
except Exception:
    pass
return _golden_section(lambda t: _nll_for_temperature(logits, y, t), lo, hi)

Three details matter:

• One parameter per label. The model is multi-label, and Cardiomegaly and Effusion have very different prevalence. A shared temperature would force one calibration curve onto two different score distributions.
• Degenerate input returns T = 1.0. If a label has only one class in the fitting split, or all logits are identical, the optimizer has nothing meaningful to fit. Returning 1.0 means "leave the model alone" rather than inventing a correction.
• SciPy is optional at runtime. If minimize_scalar is unavailable or fails, a golden-section search with the same bounds is used. That keeps the module testable in a minimal environment.

The monotone property is not just a mathematical footnote. It is the reason the A/B and C/D arms have identical decisions at the same threshold. It also means AUROC and AUPRC are exactly invariant to temperature, which the test suite asserts rather than assumes.

Deep dive: threshold policies as deterministic search

The threshold module implements five policies, all fitted on validation:

• fixed: τ = 0.50.
• f1_optimal: the threshold that maximizes F1.
• youden: the threshold that maximizes Youden's J = sensitivity + specificity − 1.
• sensitivity_constrained: the largest threshold with sensitivity ≥ 0.90, which maximizes specificity subject to that recall floor.
• precision_constrained: the highest-recall threshold with precision ≥ 0.50, with a documented fallback if the target is unreachable.

The F1 policy is short enough to read as a specification:

# src/thresholds.py
def threshold_f1(y_true: np.ndarray, y_prob: np.ndarray) -> Dict[str, object]:
    """τ maximising F1; ties broken towards the larger τ (fewer false positives)."""
    y_true, y_prob = _as1d(y_true), _as1d(y_prob)
    best = {"threshold": 0.5, "f1": -1.0}
    for tau in _candidates(y_true, y_prob):
        m = binary_metrics(y_true, y_prob, float(tau))
        if m["f1"] > best["f1"] + 1e-12 or (
            abs(m["f1"] - best["f1"]) <= 1e-12 and tau > best["threshold"]
        ):
            best = {"threshold": float(tau), "f1": m["f1"]}
    return {"threshold": best["threshold"], "constraint_met": None, "target": None}

The candidate set is the set of observed score values, not an arbitrary grid. That guarantees the chosen threshold reproduces a real confusion matrix from the data. Ties are broken toward the larger threshold, which favors fewer false positives. The constrained policies use the same candidate set but select the most specific feasible threshold or the highest-recall feasible threshold.

This design makes the policy outcomes easy to audit. For example, the sensitivity-constrained policy reaches validation recall 0.9005 for Cardiomegaly and 0.9003 for Effusion, and on test it transfers to recall 0.9205 and 0.8908. The precision-constrained policy reaches validation precision 0.5000 for both labels and transfers to test precision 0.5303 and 0.5157. The transfer is not perfect, but it is measured and reported.

The constrained policies also illustrate a trade-off that a single F1 number hides. A rule-out workflow may prefer high sensitivity even at the cost of many false positives. A referral workflow may prefer high precision even at the cost of missing cases. The repository does not declare one policy universally best; it reports the operating characteristics of each so the choice can be made explicitly.

APIs, integrations, and services

There is no HTTP API in this project. The public interface is a set of module-level command-line entry points plus the Gradio app. The main commands are:

The only external integration is the CheXpert pipeline. It looks for valid.csv and images/ under data/raw/chexpert/, then under $CHEXPERT_ROOT or $CHEXPERT_DATA_DIR, then under /mnt/data/chexpert and /data/chexpert. The label mapping is explicit: NIH Effusion maps to CheXpert Pleural Effusion. The default uncertainty policy is u_zeroes, which maps CheXpert's -1 uncertain label to absent.

Because the external cohort is absent, the pipeline writes only a status file with no_metrics_were_computed: true. There is no fabricated external metric, prediction file, or plot. That behavior is enforced by tests.

Local setup and installation

The documented environment uses Python 3.14.7 and a project virtual environment. The system Python on the original machine did not have PyTorch, so every command in the repository uses .venv/bin/python explicitly.

Prerequisites

• A recent Python 3 environment.
• A virtual environment tool (venv).
• Optional: a CUDA-capable GPU for faster inference and Grad-CAM. CPU inference works, but is slower.
• The NIH ChestX-ray14 images and Data_Entry_2017.csv if you want to re-run inference or training. The repository's research artifacts can be inspected without the raw dataset.

Clone and install

git clone https://github.com/vishalkumarojha/Heart-imaging-modelilty.git
cd Heart-imaging-modelilty

python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

For the dashboard, install the demo requirements on top of the base requirements:

pip install -r requirements_demo.txt

Data layout

Place the NIH data under data/raw/:

data/raw/
├── Data_Entry_2017.csv
├── BBox_List_2017.csv          # optional, used by Grad-CAM localization
└── images_001/ ... images_012/

The processed split cache is data/processed/split_index.csv. If it is missing, the dataset code will rebuild it from the metadata and images, but the research artifacts in this repository were produced with the frozen split.

Verify the frozen artifacts

.venv/bin/python -m unittest discover -s tests

The current checkout runs 217 tests. The frozen snapshot in outputs/research_snapshot.json records 193 tests at the time the snapshot was created; the difference reflects tests added after that freeze. The suite is designed to validate artifacts and pure functions without loading the model or touching the GPU.

Reproduce the research pipeline

# 1. Cache raw logits and probabilities
.venv/bin/python -m src.inference --split test
.venv/bin/python -m src.inference --split val

# 2. Evaluate the frozen baseline
.venv/bin/python -m src.baseline_eval --split test
.venv/bin/python -m src.baseline_eval --split val

# 3. Fit temperatures and thresholds on validation only
.venv/bin/python -m src.fit_parameters

# 4. Run experiments 1-4
.venv/bin/python -m src.experiments --exp all

# 5. Error analysis and Grad-CAM
.venv/bin/python -m src.error_analysis --split test
.venv/bin/python -m src.gradcam

# 6. External validation status (expected: blocked until data exists)
.venv/bin/python -m src.external_eval --status

# 7. Paper-ready evidence
.venv/bin/python -m src.master_results
.venv/bin/python -m src.uncertainty
.venv/bin/python -m src.paper_artifacts
.venv/bin/python -m src.paper_build

Launch the dashboard

.venv/bin/python research_demo_app.py

Open http://127.0.0.1:7860. To use another port:

GRADIO_SERVER_PORT=7861 .venv/bin/python research_demo_app.py

The dashboard binds to localhost by default. It does not require GitHub or an external service.

Environment variables

The project does not load a .env file. Set variables in the shell when needed:

# Optional: point external validation at a CheXpert mount
export CHEXPERT_ROOT=/path/to/chexpert
export CHEXPERT_DATA_DIR=/path/to/chexpert

# Optional: change the Gradio port
export GRADIO_SERVER_PORT=7861

Troubleshooting

• ModuleNotFoundError: torch: use .venv/bin/python, not the system python3.
• Live inference says the checkpoint is unavailable: the artifact tabs still work. Place the frozen checkpoint at outputs/checkpoints/baseline/densenet121_best.pt if you have it.
• Raw data missing: inference and training need the NIH images. The stored research artifacts can still be inspected and many tests still run.
• pytest not found: the project uses unittest; run python -m unittest discover -s tests.
• External validation is blocked: that is the expected state until a CheXpert cohort is present. The status file explains every path that was probed.
• Gradio port already in use: set GRADIO_SERVER_PORT to a free port.
• Small differences after recomputing inference: AMP fp16 can flip a few rows near 0.50. The project uses one cached prediction table for all experiments so the difference does not compound.

Testing and reliability

The test suite is not a token smoke test. It checks:

• the frozen checkpoint hashes and the presence of the legacy root checkpoint;
• the patient-level split integrity and documented row counts;
• that published metrics recompute from the stored prediction tables;
• that calibration and threshold fitting refuse evaluation splits;
• that AUROC and AUPRC are invariant under temperature scaling;
• that the Grad-CAM case list is deterministic and every case image exists;
• that the external phase remains honest while the dataset is absent;
• that the paper tables, figures, and claim audit agree with their source artifacts.

Reliability mechanisms in the code include:

• Atomic JSON writes (tmp file + replace) so a crash does not leave a truncated result.
• NaN for undefined metrics instead of raising or silently returning zero.
• Actionable FileNotFoundError messages that name the missing artifact and the command that produces it.
• Runtime guards on validation-only fitting.
• Hash-keyed prediction caches so a checkpoint change invalidates old predictions.
• Deterministic case selection for Grad-CAM, ordered by certainty and image index with no RNG.

There is no CI workflow in the repository. The tests are run locally. The frozen snapshot records a passing run, and the current checkout passes when run with .venv/bin/python -m unittest discover -s tests.

Security and privacy considerations

This is a local research tool, not a deployed clinical service. The security posture is correspondingly simple:

• The Gradio app binds to 127.0.0.1, not to a public interface.
• Uploaded images are processed locally; the app does not send them to an external API.
• There are no API keys, tokens, or secrets in the repository.
• The external CheXpert integration reads a local filesystem path; it does not authenticate to a remote service.
• The research artifacts contain image indices and patient IDs from a public, de-identified dataset.

Those points are not a claim that the system is "secure" in a clinical sense. If the dashboard were exposed beyond localhost, it would need authentication, TLS, access control, audit logging, retention policies, and a careful review of any protected health information. The repository explicitly labels the system as a research demonstration and not clinical software.

Deployment and infrastructure

The current deployment model is a local workstation. The dashboard is a Gradio process launched from the repository root. The model checkpoint, data, and result artifacts are local files. The raw dataset and checkpoints are git-ignored because they are large; a fresh clone needs them provisioned separately to reproduce inference.

There is no Dockerfile, no Kubernetes manifest, no cloud infrastructure, and no CI/CD pipeline in the repository. That is an honest description of the current state, not an omission in this article. The project was built to produce a reproducible research package on a single machine, and the local dashboard reflects that.

A future deployment could containerize the dashboard, put it behind an authenticated reverse proxy, store artifacts in an object store, and add a CI job that runs the test suite and verifies the frozen hashes. Those are proposed improvements, not existing infrastructure.

Engineering challenges and trade-offs

Frozen model vs. flexibility. Freezing the checkpoint makes the calibration and threshold comparison clean, but it also means the project cannot explore whether a different backbone or training recipe would change the conclusions. The trade-off is deliberate: the research question is about the decision layer, not the model.

Validation-only fitting vs. data efficiency. Fitting temperatures and thresholds on validation only is the right experimental discipline, but it leaves less data for parameter estimation. The constrained policies can miss their targets on validation, and the project reports that rather than tuning on test.

NLL vs. ECE. Temperature scaling improves NLL and Brier while raising binned ECE. The project could have optimized a different objective or chosen a different calibration method, but the honest result is more useful than a metric that only goes in the desired direction. The repository keeps both numbers visible.

F1-optimal vs. clinical operating points. F1-optimal is a reasonable default for a balanced summary, but it is not a clinical policy. The constrained sensitivity and precision policies show the trade-offs, and the project does not claim that one is universally correct.

Image-level vs. patient-level statistics. The older confidence-interval pass resampled test images. The upgraded analysis resamples patients, which is the statistically defensible unit because images from one patient are correlated. Both artifacts exist; the legacy one is labeled as legacy, and the primary endpoint uses the patient-level bootstrap.

External validity. The external pipeline is implemented, but the cohort is absent. The project refuses to estimate external performance. That is a limitation, but it is also a strength: the status file says exactly what is missing and how to unblock it.

Explainability. Grad-CAM is qualitative. The repository adds a quantitative sanity check on a fixed cohort of 43 test images with NIH bounding boxes: the pointing game hits 21/43 (48.8%), and the heat-mass concentration ratio is 2.60 on average, with 38/43 cases above 1. That is evidence that the heatmaps are not random, but it is not a validation of clinical localization. The article and the dashboard both keep that caveat.

Current limitations and future improvements

Verified limitations:

• One patient-level split, one seed, one training run. There is no cross-validation.
• Only two of the fourteen NIH findings are modeled.
• The local image collection is partial: 109,312 images are available from the 112,120 metadata rows, so 2,808 images are missing.
• NIH labels are report-mined, not radiologist-adjudicated for this study. Label noise is a real possibility.
• The split is the project's own patient-level 70/15/15, not the official NIH split lists. Absolute numbers are not directly comparable to papers that use the official split.
• External validation is pending; no external metrics exist.
• Binned ECE rises after temperature scaling, even though NLL and Brier improve.
• The legacy image-level bootstrap remains in the repository alongside the patient-level upgrade.
• Grad-CAM localization is a small, fixed cohort sanity check, not a clinical validation.
• There is no CI pipeline, no container image, and no public deployment.
• The dashboard is read-only over precomputed artifacts; it does not support live research runs.

Reasonable future improvements:

• Add an external cohort and run the already-implemented CheXpert evaluation with the frozen model and thresholds.
• Add multi-seed or cross-validated training runs if the research question expands beyond the frozen-control design.
• Extend the label set beyond Cardiomegaly and Effusion.
• Compare temperature scaling with Platt scaling, isotonic regression, or beta calibration under the same validation-only protocol.
• Add decision-curve analysis and cost-sensitive operating points for specific workflows.
• Add CI that runs the test suite, verifies checkpoint hashes, and renders the Mermaid diagrams.
• Containerize the dashboard and add authentication, TLS, and audit logging before exposing it beyond localhost.
• Add live Grad-CAM with a clear non-clinical caveat, or keep it as a saved exploratory artifact.

What developers can learn from this project

• Separate ranking from deciding. AUROC and AUPRC measure discrimination; calibration and thresholds determine decisions. A model can be good at one and bad at the other.
• Monotone transforms preserve ranking but change probability scale. Temperature scaling cannot improve AUROC, and it cannot change decisions at a fixed 0.50 threshold. If you change the probability space, refit the threshold in that space.
• Fit parameters on validation and enforce it in code. A documentation rule is easy to break. A runtime ValueError plus tests is harder to ignore.
• Cache the model outputs, not just the metrics. Storing logits lets you re-derive calibrated probabilities without re-running the model and keeps every experiment on the same predictions.
• Treat artifacts as the source of truth. JSON for canonical results, CSV for tidy tables, PNGs for derived views. Tests can then recompute published numbers from the same files.
• Use the right resampling unit. Patient-level clustering matters in medical imaging. A patient-level bootstrap is more defensible than resampling images independently.
• Report the metric that got worse. The ECE increase after temperature scaling is more informative than a selectively reported improvement.
• Build the demo on frozen evidence. A dashboard that reads verified artifacts is easier to trust and easier to test than a dashboard that recomputes research results on every click.

Conclusion

The repository starts from a familiar situation: a chest X-ray classifier with good ranking metrics and no clear decision policy. It responds with a controlled study of the decision layer. A frozen DenseNet121 is evaluated at a fixed baseline, its logits are cached once, temperature scaling and five threshold policies are fitted on validation only, and the resulting operating points are evaluated on held-out test data. The A/B/C/D decomposition shows that threshold selection changes F1, while temperature scaling improves NLL and Brier without changing ranking or fixed-threshold decisions. The project also reports the uncomfortable result: binned ECE rises after calibration.

The engineering value is in the discipline around the numbers: one checkpoint, one prediction cache, validation-only fitting guards, patient-level statistical inference, deterministic error and Grad-CAM case selection, artifact-level tests, and a dashboard that reads the verified evidence instead of inventing a second version of it. The external-validation question remains open, and the repository says so explicitly.

If you are building a model-evaluation pipeline, the most reusable idea here is not a particular calibration method. It is the separation of parameters, results, and presentation — and the willingness to report a metric that moves in the wrong direction when the evidence says it does.

References and further reading

• Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017). ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. CVPR. arXiv:1705.02315.
• Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. CVPR. arXiv:1608.06993.
• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML. arXiv:1706.04599.
• DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837–845.
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
• Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV. arXiv:1610.02391.
• Irvin, J., et al. (2019). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. AAAI. arXiv:1901.07031.

Project links

• Source code: github.com/vishalkumarojha/Heart-imaging-modelilty
• Research dashboard: local only; launch with python research_demo_app.py
• Documentation: the repository's docs/ directory contains the scope, baseline, experiment, reproducibility, and status reports
• External validation: not available; see outputs/metrics/external/status.json