APPENDIX · P6 · REPEAT INSERTION · FOR SLIDE 18
← Back to Slide 18
P6. Deduplication compares blocks
INSERT INTO kills_raw_dedup
VALUES (888, 1042, 9); -- та же строка, другая пачка
SELECT sum(kills) FROM kills_total_dedup WHERE player_id = 1042;
INSERT INTO kills_raw_dedup SETTINGS insert_deduplication_token = 'match-889'
VALUES (889, 1042, 2);
INSERT INTO kills_raw_dedup SETTINGS insert_deduplication_token = 'match-889'
VALUES (889, 1042, 2);
SELECT sum(kills) FROM kills_total_dedup WHERE player_id = 1042;
What the result should be
18 // другая пачка — новый блок:
// 9 + 9
20 // 18 + 2: второй INSERT
// с тем же токеном отброшен
Block Hash
Bundle token
Where does the ID come from?
The same line in a batch with a different composition has a different hash: the murders were counted a second time.
`insert_deduplication_token` replaces the hash: duplicates with the same ID are discarded, resulting in 20 instead of 22.
Consistent across repetitions: outbox message ID, queue offset, match number, and batch number.
Running P6 again won't do anything: both blocks are already in the window. To start over, recreate your database (reset in the results window).