Datacurator — dataset pipeline

Problem it solves

Every style LoRA on this site started as a dataset problem, and dataset quality is the single highest-leverage variable in style training — a rank-32 LoRA on 58 images has no capacity to ignore bad data. Collecting references, de-duplicating, cropping, rejecting off-topic images, captioning and exporting in trainer format is a full pipeline’s worth of work, so I built it as one.

The pipeline

Crawl → ML-assisted selection → human review → caption → export.

1. Crawl. Keyword- and gallery-based collection from reference sources.
2. ML-assisted selection. Perceptual de-duplication, aesthetic/quality scoring, and CLIP-embedding similarity suppression so the kept set stays diverse instead of collapsing onto the artist’s ten most repeated compositions. Rule-based auto-rejects (multi-panel sheets, screenshots, group shots) run as a natural-language instruction filter through a vision-language model.
3. Human review queue. Everything the machine kept still passes a keep/reject pass — for the flat-casual set that meant 1,167 images reviewed → 521 kept → 58 promoted to the final training set.
4. Caption engines. Switchable back-ends: booru-style taggers for tag-based models, VLM prose captioning for natural-language models — with a caption viewer for manual QA, because caption hallucinations (weapons and wings that aren’t there) poison style training.
5. Export. One click produces the trainer-ready folder: images, captions, and the repeat structure the training config expects.

Thumbnails blurred: source artworks are used for analysis during training-data curation and are not redistributed.

Why it exists

“Train a LoRA” sounds like a GPU task; in practice it is 80% data discipline. Making the data path a repeatable pipeline is what lets a new style go from idea to validated model in a day or two — and what makes the resulting models consistent enough to put into a production pipeline.