Jass Card Detection ML Setup Guide

This document outlines the strategy and implementation details for transitioning the card detection system from basic color thresholding to a deep learning-based object recognition pipeline.

1. Architecture Overview

The system uses a Hybrid Pipeline to balance performance on mobile devices with high accuracy.

Workflow: Camera Feed \rightarrow Image Processing (Localization) \rightarrow Card Cropping \rightarrow ML Classifier (Identity) \rightarrow Game State Update

Localization: Uses brightness and shape analysis to identify bounding boxes of cards (existing logic in Detection.tsx).
Classification: Crops each detected card and passes it through two specialized TensorFlow.js models to determine the Suit and the Value.

2. Data Collection & Labeling

Because Jass cards have unique iconography, we use a custom dataset created from internet samples.

Labeling Strategy: Folder-Based Annotation

Instead of using bounding-box tools, we use the directory structure as labels.

Directory Structure:

dataset/
├── suit_model/
│   ├── Schellen/  (Bells)
│   ├── Schilten/  (Shields)
│   ├── Eicheln/   (Acorns)
│   └── Rosen/     (Roses)
└── value_model/
    ├── 6/
    ├── 7/
    ...
    └── 13/

Process:

Sourcing: Collect 30-50 images per class from Swiss-German Jass deck galleries.
Cropping: Crop the center of the card for the Suit model and the corners for the Value model.
Augmentation: Use a script to generate variations:
- Rotations (\pm 15^\circ)
- Brightness/Contrast shifts
- Gaussian noise to simulate mobile camera grain.

3. Model Training (Python)

We use Transfer Learning to minimize the required dataset size.

Base Model: MobileNetV2 (pre-trained on ImageNet).
Modification: Remove the final 1000-class layer and replace it with a Dense layer matching the number of labels (4 for suits, 8 for values).
Optimization:
- Loss: categorical_crossentropy
- Optimizer: Adam
- Regularization: Dropout (0.2) to prevent overfitting.
Target Size: 64x64 pixels.

4. Conversion & Deployment

Once trained in Python, the models are converted to the TensorFlow.js format.

Conversion Command:

tensorflowjs_converter --input_format=keras /path/to/model.h5 /public/models/suit_model

Deployment Path: The app expects the models in the public directory:

/public/models/suit_model/model.json
/public/models/value_model/model.json

5. Frontend Integration

The integration is handled by src/services/CardModelService.ts.

Initialization: Models are loaded in parallel during App.jsx mount.
Inference:
- Tensors are created from the canvas crop.
- Normalization is applied (div(255.0)).
- tf.tidy() is used to wrap operations and prevent WebGL memory leaks.
Fallback: If models are missing or fail to load, the app automatically reverts to the legacy color-analysis detection to ensure the app remains functional.

6. Validation Metrics

To verify the setup, the following metrics should be measured:

Inference Latency: Time from "Scan Table" click to result (Target: < 500ms).
Classification Accuracy: Percentage of correct identifications on a hold-out test set.
Memory Footprint: GPU memory usage during scan.

3.4 KiB Raw Blame History