workflow.log - master - maratgaliulin/landcode_classifier

workflow.log
236 строк10 KB
maratgaliulin
23 июн 2026, 15:04
23 июн 2026, 15:04488c495
# LandCodeClassifier — Workflow Log

Purpose: running record of project decisions, context, and Cursor conversation summaries.
Do NOT duplicate git history here — use git / gitverse for commits and diffs.

How to use:
  - Ask the agent: "Add today's workflow to workflow.log" (or similar).
  - Each entry should capture: goal, what was discussed/decided, outcomes, open questions, next steps.
  - Keep summaries concise; link to files or scripts when relevant.

Format:
  ================================================================================
  YYYY-MM-DD | Session summary
  ================================================================================
  Topic:
  Context:
  Discussion / decisions:
  Outcomes:
  Open questions:
  Next steps:
  ================================================================================


================================================================================
2026-06-22 | Session summary
================================================================================
Topic:
  Recovering project work history after switching to a new computer.

Context:
  Project was developed in Cursor on another machine. Chat/agent history does not
  sync across devices; only local Cursor data on the original PC holds past
  conversations.

Discussion / decisions:
  - Git history on gitverse is the source of truth for code changes — no need to
    copy it into this log.
  - Cursor chats live locally per machine (%APPDATA%\Cursor\User\workspaceStorage,
    ~/.cursor/projects/.../agent-transcripts/). This machine had almost no prior
    chat history for LandCodeClassifier.
  - To recover old AI context: copy/sync Cursor folders from the other PC, or use
    tools like cursaves / Cursor Chat Transfer extension.
  - Created this workflow.log to record future conversation summaries instead of
    relying on cross-machine Cursor history.

Outcomes:
  - workflow.log established as the project journal for session summaries.

Open questions:
  - Is access still available to the other computer for exporting Cursor chats?

Next steps:
  - After each significant Cursor session, ask the agent to append a summary here.
  - Optionally export chats from the old machine if full conversation text is needed.

================================================================================


**Operating as Architect — recommendation only.**

After the mapping fix, the bottleneck is clearly the model: **45.4%** on 346,873 rows (87.7% of data), while mapping is **98.3%** on the remaining 12.3%. Overall accuracy is **51.9%**. Below are suggestions ranked by expected impact on model accuracy.

---

## 1. Align training with inference (highest impact)

This is the biggest structural issue in the codebase.

| Stage | Current behavior |
|-------|------------------|
| **Training** (`train_new_model`) | `BertWithNumeric` — text only, no cluster embedding |
| **Inference** (`OKSPredictorWithCluster`) | `BertWithNumericAndCluster` — text + `area_cluster` embedding |

Training also uses `BertLandUseDataset` (no clusters), while inference passes `cluster_ids`. The cluster embedding layer is never trained properly; weights may be random or mismatched.

**Recommendation:** Retrain with:
- `BertWithNumericAndCluster`
- `BertOKSDatasetWithCluster` (pass `area_cluster` from pickle)
- Same `num_clusters=6` as at inference

This alone can give a noticeable lift because the model will actually learn the area signal you already engineer in `combined_text` (`кластер_N`).

---

## 2. Fix train/infer preprocessing consistency

Small mismatches add up across 346k rows.

**`combined_text` casing**

- Training pickle: `.str.lower()` in `return_single_dataframe_from_pickle_or_from_excel_with_clusters`
- Inference: `preprocess_dataframe` joins columns **without** `.str.lower()`:

```599:599:methods/classes/OKSPredictorWithCluster.py
            df_processed['combined_text'] = df_processed[text_cols_with_cluster].fillna('').astype(str).agg(' | '.join, axis=1)
```

**Clusterizer for long model**

- Long model uses `clusterizer_dir` (standard), not `clusterizer_dir_long` in `train_oks_model.py` predict branch — cluster IDs at inference may not match training distribution.

**Recommendation:** Use identical preprocessing for train and predict: lowercase, same clusterizer per model variant, same column list from checkpoint `text_columns`.

---

## 3. Use the best checkpoint, not the last epoch

`train_new_model` saves both:
- `*_best.pth` — lowest validation loss
- final `.pth` — epoch 15

Inference loads the non-`_best` path (`rubert_landuse_model_oks_long.pth`). The last epoch is often worse than the best validation checkpoint.

**Recommendation:** Load `*_best.pth` / `*_best.pkl` in `OKSPredictorWithCluster`, or copy best weights to the default path after training.

---

## 4. Handle class imbalance (214 classes)

Rare subgroup codes dominate errors. Current setup:

- Plain `CrossEntropyLoss`
- `train_test_split` **without** `stratify` (commented out at line 629 in `utils.py`)
- `FocalLoss` exists but is unused

**Recommendations (pick 1–2 first):**

| Technique | Effect |
|-----------|--------|
| **Stratified split** | More reliable validation; better early stopping |
| **Class weights** in `CrossEntropyLoss` | Upweight rare codes |
| **`FocalLoss`** (γ=2) | Focus on hard examples |
| **Minimum samples per class** | Drop or merge classes with &lt; N examples |
| **Balanced accuracy / macro-F1** for model selection | Don't optimize raw accuracy on skewed data |

For cadastral codes, **balanced accuracy** and **per-class recall on rare codes** are better metrics than overall accuracy.

---

## 5. Improve the training recipe

Current hyperparameters in `train_new_model`:

- `lr=2e-5`, `batch_size=16`, `15 epochs`
- No scheduler, no weight decay, no gradient clipping
- Full BERT unfrozen from the start

**Suggested experiments:**

1. **Discriminative learning rates** — lower LR for BERT body (1e-5), higher for classifier head (1e-4).
2. **Linear warmup + decay** — standard for BERT fine-tuning.
3. **Weight decay** — `AdamW` with `weight_decay=0.01`.
4. **Early stopping** — `SmartEarlyStopping` is already in the repo but unused; stop when val loss plateaus.
5. **Longer sequences** — long model joins 9 text fields; `max_length=256` truncates heavily. Try 384 or 512 if GPU memory allows.
6. **More epochs with early stopping** — 15 fixed epochs may under- or over-fit.

---

## 6. Label and target design

Ground-truth values look like full strings: `"0406 Многоэтажный жилой дом (МКД)..."`, not just `"0406"`.

**Issues:**
- 214 classes with long label strings — harder than classifying code prefixes
- Log shows **175 unique codes** without group mapping — label space may be noisy

**Recommendations:**
- Train on **code prefix only** (`split()[0]`) if business metric is code, not full description.
- Or keep full string but ensure train labels and evaluation column use the **same format**.
- Audit confusions between codes that share text (e.g. `1031_1` vs `1031_2`).

---

## 7. Inference strategy (not retraining, but raises effective accuracy)

Even a 45% model can be used more safely:

| Strategy | Idea |
|----------|------|
| **Confidence threshold** | If `prediction_confidence < 0.4`, flag for manual review instead of auto-assigning |
| **Top-3 suggestions** | `get_top_predictions` already exists — surface alternatives for low-confidence rows |
| **Hybrid routing** | Mapping first (98% accurate), model only for unmapped — already in place |
| **Expand exact mapping** (variant B from recommendations doc) | Shifts easy rows off the model; model focuses on harder 75% |

Expanding safe exact mapping is often cheaper than pushing model accuracy by 10+ points.

---

## 8. Feature and data quality

Long-model `text_columns_long` includes material, floors, etc., but mapping only uses `Наименование объекта`. The model should leverage all columns — verify they are non-empty at inference and included in `combined_text`.

**Recommendations:**
- Drop or impute rows where all text columns are empty before prediction.
- Add `Назначение здания` / `Назначение сооружения` as separate mapping priorities (variant B) — reduces hard cases for the model.
- Refresh pickle cache after Excel changes (stale `df_oks_long.pkl` = stale training data).

---

## 9. Evaluation before next training run

To prioritize fixes, analyze errors on the 346k model-predicted rows:

1. **Confusion matrix** — top 20 confused code pairs.
2. **Accuracy vs confidence** — likely &lt;30% below 40% confidence, much higher above 70%.
3. **Accuracy by class frequency** — frequent classes vs rare.
4. **Truncation rate** — share of samples where `combined_text` exceeds 256 tokens.

`ModelEvaluator` and `ABTestRunner` already exist for this.

---

## Suggested implementation order

| Priority | Action | Expected gain |
|----------|--------|---------------|
| 1 | Retrain with `BertWithNumericAndCluster` + cluster dataset | High |
| 2 | Load `*_best.pth`; fix lowercase + clusterizer mismatch | Medium |
| 3 | Stratified split + class weights or FocalLoss | Medium |
| 4 | LR schedule + early stopping | Medium |
| 5 | Code-only labels (if metric allows) | Medium–high |
| 6 | Increase `max_length` / audit truncation | Medium |
| 7 | Confidence gating at inference | Effective accuracy ↑ without retrain |

---

## Realistic expectations

- Mapping fix already raised overall accuracy from **27.6% → 51.9%**.
- Model-only accuracy went from **34% → 45.4%** (partial mapping had been polluting model evaluation too).
- Getting model accuracy from **45% → 60%+** on this 214-class, text-heavy task will likely need **items 1–5 together**, not a single hyperparameter tweak.
- Getting toward **80%+ overall** will also require **expanding exact mapping coverage** so the model handles fewer rows.

If you want this implemented, switch to **Agent mode** and confirm which items to start with (I'd recommend **1 + 2 + 3** as the first training pass).

[REDACTED]