Canine EEG validation before decoding.
This project checks whether a canine EEG model is learning condition structure, or just picking up file import errors, label uncertainty, recording artifacts, and dog identity.
The classifier is used as a validation probe, not as the main result. Before reading the score, I check whether the signal import, event structure, label provenance, QC choices, dog-level splits, and null comparison are strong enough to support interpretation. In the current run, leave-one-dog-out decoding gives macro AUC around 0.503. Since 0.5 is chance level for AUC, this says the current candidate labels and features do not separate the conditions beyond the null.
1. The small N problem
The downstream analysis has 2,820 epochs from 16 dogs. Those epochs are treated as repeated measurements, not as 2,820 independent samples.
That is why the split matters. With mixed dog splits, decoding can pick up dog identity, session structure, baseline differences, or preprocessing artifacts. Leave-one-dog-out evaluation keeps the test dog unseen during training.
Unit
The dog is the independent subject. Epochs improve measurement density, not sample size.
Split
Leave-one-dog-out evaluation keeps the test dog unseen during training.
Readout
AUC is compared with a dog preserving shuffled label reference.
2. Validation checks before decoding
Before modeling, the dataset itself becomes the object of validation. The question is not which classifier performs best, but whether the inputs can support a decoding claim at all.
Read the CNT files correctly.
The first failure was file reading. A previous import path treated 32-bit CNT samples as split 16-bit words. After fixing that, all 17 recordings pass the 32-bit import check, with checksum agreement and maximum MNE/header scaling error of 2.91e-11 uV.
Separate trigger quality from condition meaning.
Trigger codes can have the right counts and order even when the stimulus mapping is incomplete. These two questions are kept separate: whether the event stream is internally consistent, and whether a code can be trusted as a condition label.
Keep threshold choices visible.
Recording and epoch metrics are stored together with the threshold status. Some thresholds are exploratory, and they should be marked that way. Human EEG QC rules are a reference point, not an automatic fit for canine recordings.
Look for dog and session structure before interpreting labels.
The analysis checks whether embeddings, a 3-axis PCA of the same 88 features used only for this confound visualization and not by the decoder, carry dog identity. It also checks whether session position tracks labels, and whether baseline activity persists into the response window. If that structure is present, decoding can reflect the recording setup before it reflects the condition.
Use dog level splits and a dog preserving null.
With candidate condition labels, leave-one-dog-out macro AUC is 0.5034. The within dog shuffled label null (20 permutations per dog) averages 0.4963, SD 0.0091, 95th percentile 0.5095; no dog fold beats its own null p95. Since 0.5 is chance level for AUC, this result says the current labels and features do not support a condition decoding claim.
Record enough to rerun the analysis.
Each run stores raw checksums, the config hash, software versions, random seed, exact command, and report checksums. That makes the result traceable to the files and settings that produced it.
3. Dataset
The case study uses the public canine EEG data from Magyari et al. (2020), an auditory word processing experiment in dogs. The raw data include 17 numeric Neuroscan CNT recordings. Each recording is expected to contain eight channels sampled at 1000 Hz. A complete recording has trigger codes 1-48, five repeats per code, and 240 stimulus events.
The downstream set has 16 dogs and 2,820 epochs. Dog 40 is excluded because of QC and event integrity concerns. The archive is linked so the data source is explicit.
Public Figshare archive for the recordings used in Magyari, Huszar, Turzo, and Andics (2020).
Open Figshare dataset Open source paper4. Method
The analysis starts with the data, not the classifier. I verify the CNT import, event stream, candidate label mapping, QC choices, and subject structure first, because each of those can change what a decoding score means.
Decoding is then run with leave-one-dog-out splits. The held out dog is never seen during training, and macro AUC is compared with a within dog shuffled label null. This keeps the evaluation focused on subject independent signal instead of repeated epochs from the same dog.
Each epoch is reduced to 88 features: 8 channels times 11 measures each (five band powers, three Hjorth parameters, line length, mean, variance). A scaled logistic regression is fit per fold with hyperparameters fixed in advance, not tuned on the held out dog. The null comes from 20 within dog label permutations per dog, so each dog gets its own null distribution instead of one pooled number.
5. Results
The AUC number is not the main output here. The CNT import bug is fixed and checksum verified, event issues are logged per recording instead of pooled over, QC thresholds are stated instead of assumed, confounds are checked directly, and dog level decoding has an actual null distribution to compare against.
| Result | Evidence | Interpretation |
|---|---|---|
| CNT decoding | 17/17 files show corrected 32-bit consistency and checksum agreement. | The raw signal path is now traceable and checkable. |
| Event integrity | 9 recordings match the exact event design; 8 require event review. | Recording level event issues are visible before any pooled analysis. |
| Condition mapping | The original trigger code to stimulus codebook was not recovered locally. | The analysis keeps candidate labels useful without overstating them as verified conditions. |
| Dog level decoding | Macro AUC is 0.5034 (88 band power / Hjorth features, logistic regression, leave-one-dog-out). Within dog shuffled null: mean 0.4963, SD 0.0091, p95 0.5095 across 20 permutations per dog; 0 of 16 dog folds exceed their own null p95. | The result gives a calibrated baseline for future labels, features, and larger dog cohorts. |
| Reproducibility | Checksums, config hash, software versions, command, seed, and report outputs are recorded. | The run can be inspected, rerun, and compared with future analyses. |
6. Limitations and open questions
The near null result should be read narrowly. It does not say that the EEG has no information. It says this label set, feature set, and leave-one-dog-out setup did not separate the candidate conditions beyond the dog preserving null.
The biggest open issue is still the condition mapping. The original trigger code to stimulus codebook was not recovered locally, so the labels stay candidate labels. The other limitation is sample size at dog level. More dogs and verified labels could change the result; they just need to be tested with the same dog held out discipline.
7. Conclusion
The next step is to test the same checks on data with cleaner metadata: verified condition mapping, more dogs if possible, transparent QC, and the same leave-one-dog-out evaluation. A larger classifier only makes sense after that.
The current dataset is enough to build and test the validation pipeline, but not enough for broad claims. Because comparable canine EEG recordings are not easy to find, I have contacted a couple of other authors about related data. A second dataset would make it possible to test whether the same import, label, QC, and subject split issues appear again.