Six defects, and how each one surfaced
Five of these are PIGEON’s own bugs, not a competitor’s. The sixth is not a bug at all — it is an episode that changed what this project accepts as evidence, which is why it sits with the others. They are here because a transfer tool asks to be trusted with the one thing its user cannot check by eye, and the honest way to earn that is to show the work rather than assert the result.
1. A hash cache that is allowed to miss one edit — and says so
File-sync tools have a long history of trusting file timestamps, and the reason is obvious: hashing a large tree costs real time, and size-plus-modification-time is free. It is also defeatable. Any cache keyed on size and timestamp can be handed a file whose content changed and whose timestamp did not.
PIGEON hashes by content and has no timestamp shortcut at all unless a profile opts into one. When it does, the cache carries two independent guards.
The first is the rule git uses against the same problem. A cached entry is trusted only when the file’s modification time is strictly older than the cache’s own, so an edit landing in the same second as the run that cached it is never trusted and the file is re-hashed. A later pass tightened the reference point: the threshold is the moment the scan started, not the moment the cache was written. A file edited during a long scan — after its hash was taken, before the cache was saved — would otherwise have a timestamp older than the cache file and be trusted.
The second guard exists because the first cannot be made sufficient. Content changed with the modification time deliberately restored is invisible to any rule founded on that timestamp. Not to a stricter one; to all of them. So there is a periodic full rescan, on a schedule the profile sets, that ignores the cache entirely.
The test suite demonstrates that miss on purpose. It makes a same-length edit, restores the file’s modification time bit-for-bit, and asserts that the run reports a cache hit and does not deliver the change — then forces the rescan due and shows the same edit being found. The note beside it says not to “fix” that section by tightening the first rule, because no timestamp rule can see that edit and the honest answer is the backstop.
And a real timestamp bug this work surfaced. C++17 has no portable conversion from a filesystem timestamp to a wall-clock one. The workaround is to sample both clocks and use the difference as an offset — and the original code re-sampled both, fresh, for every file on every scan. That quietly assumes the two clocks advance in lockstep at the same resolution. On one Windows toolchain they do not, so the same unmoved filesystem timestamp converted to two different whole-second values depending on precisely when each scan happened to sample the offset. A comparison of “same size, same timestamp, different content” — a test that restores the timestamp on purpose, to prove the difference engine is not fooled by it — failed because the two timestamps no longer compared equal, though the filesystem attribute had never moved. Each clock is now sampled once for the life of the process, so the conversion is a deterministic function of the raw timestamp and nothing else.
2. A temporary filename that grew past the limit
An upload handler that writes to a staging file and renames it into place on success is the ordinary way to avoid leaving a half-written file at the real destination. The obvious way to name that staging file is to extend the name the client sent.
That is a bug, and an unsubtle one once seen. Filesystems limit the length of a single path component — 255 bytes on both ext4 and NTFS — and the suffix is around forty characters. A filename already within forty bytes of the limit produces a staging name over it.
Reproduced rather than reasoned about: a 234-byte filename, comfortably legal, confirmed to write directly with an ordinary copy, was refused by PIGEON’s own embedded HTTPS server with a 500. Two upload handlers had the identical defect — the same suffix string, apparently copied from one file to the other.
The fix stops extending the client’s name at all. The staging name is derived from a hash of the destination path, placed in the destination’s own directory so the final rename is still same-filesystem and still atomic, and its length no longer depends on the real filename’s length. Every other staging convention in the codebase was then audited: the rest key off a small integer index or a fixed internal filename, so none of them can reach the limit by construction rather than by luck.
A second finding in the same helper, several passes later. The name it built was unique only because two threads rarely read the clock on the same tick. Measured, that was 15 collisions in 200,000 names. Two concurrent writers of one file then shared a staging name, both opened it truncating, and the losing writer’s own short-write check deleted the bytes the other had just staged — a name collision in a helper whose only job is to be unique, turning into the one outcome this product exists to prevent. A timestamp is not an identity: the name now also carries the process id and an atomic counter, which are unique by construction. The regression test was run against the unfixed code first and produced 2 collisions in 32,000.
3. The hole at the end of the file
Sparse-file support is a small feature. When a chunk of the source is entirely zero bytes, seek past it at the destination instead of writing it and let the filesystem leave a hole. A hole reads back as zeros, identical to what the source held there, so the existing verification needed no changes at all.
Except that seeking past the end of a stream, with no write after it, does not extend the file when the stream closes. It is an elementary fact about C++ streams, and exactly the one you walk into when the entire point of the feature is to seek instead of writing. A source file whose last chunks are all zeros would land short at the destination — silently, and while still hashing correctly for every byte it did contain. A truncated file that verifies for everything it holds is precisely the failure this product exists to prevent, arriving through the front door of a feature meant to be an optimisation.
The fix is one unconditional resize to the source’s real length after the write loop closes the file: a correct no-op in every ordinary case, and the actual fix when the trailing chunks were holes. One call covers both without tracking which happened.
The feature also cannot be proven by comparing hashes, because a fully allocated file and a genuinely sparse one hash identically — sparseness is a storage property invisible to content. The test reads the allocated block count instead. A realistically shaped 16 MB file — real content, a 16 MB run of zeros, real content again, rather than an all-zero file that would never exercise the interesting case — allocates 16,781,312 bytes copied normally and 266,240 bytes copied sparse, with both copies byte-for-byte identical to the source and both passing verification.
NTFS needs the file marked sparse-capable before an extension can become a real hole; without that mark the identical operation is zero-filled and fully allocated, which defeats the point quietly rather than loudly. A destination that cannot do it — FAT, many network shares — still receives the file byte-correct, just fully allocated, and the run records in its own receipt that the profile’s intent did not fully happen.
4. The move that never moved
A move is a copy, a verification, and then a deletion of the source. On Windows, PIGEON’s version never performed the third step. Not intermittently — every file, every run, for as long as the feature existed.
The cause is one line that was not there. The read handle the copy opened on the source was still open when the deletion was attempted a few lines later. On POSIX, unlinking a file that is still open is entirely ordinary; the directory entry detaches and the data survives until the last handle closes. Windows enforces sharing at the handle level, and an ordinary read stream does not ask for delete sharing, so every deletion was refused.
What makes it worth telling is the shape of the failure rather than the cause. The run reported completed. The correct content landed at the destination and verified. The deletion’s own failure path did exactly what it was designed to do for the legitimate case — a read-only source, a permissions problem — which is to log a warning and increment a counter. It simply happened to be doing that for every file on every run, so a feature an operator had deliberately switched on did nothing at all, in the one direction where nothing is the opposite of what was asked.
It was found the first time the suite ran on a native Windows machine rather than on Linux or under a compatibility layer. Two apparently unrelated failures in other suites pointed at the same root cause and confirmed it: the emptied source directory was never pruned, because nothing had emptied it; and in a hardlink test only the primary group member survived, because only the primary goes through the code path that holds the read handle open.
5. A verification that verified itself
Windows permission reproduction: read the source file’s owner, group and access-control list, and apply them to the destination. The system call returned success, so the first version trusted it.
A real end-to-end run showed otherwise. The call succeeded, the run reported no degradation, and the destination file’s actual permissions were the default the destination volume would have produced anyway. So a verification step was added, in keeping with how the rest of this product treats a library’s return value: re-read the destination’s security descriptor immediately afterward and compare it, field by field, against what was requested.
The same test then reported a clean match.
Taken alone, that reads as “the earlier result was a fluke and the code is fine.” It was not. A second, genuinely separate process — sharing no state, no handle and no cache with the one that had just run the copy — read the same file and saw the unchanged original. The change had never reached the real file on disk. The read-back had been answered by something the writing process could see and nothing else could.
The cause turned out not to be PIGEON’s code. The compatibility layer the test was running under answers that pair of calls consistently to the process that made them, and does not carry the change through to the real filesystem underneath — confirmed by reproducing it with a standalone program that shared no code with PIGEON at all, rather than inferred from the symptom. Which is precisely why it belongs in this list: what the episode changed is not a line of transfer code, it is the standard of evidence.
The generalisable part is the one worth keeping: a same-process “verify what you just set” check is not evidence that a change persisted. It looks exactly like a working implementation, because a working implementation answers identically. Only a reader with no relationship to the writer can tell the two apart. That requirement was written down as binding on anything built on the same pair of calls — and when a real Windows machine later checked the access-control half from a separate process, it passed.
The honest remainder, stated here as it is stated internally: the owner half of that check has no negative control. Nothing establishes that an ordinary copy would not already have produced the same owner, so it could pass without the code having set anything. Access-control reproduction on real Windows is confirmed. Owner reproduction is not contradicted and is not proven.
6. A corrupt state file failed — but failed vaguely
The incremental air-gap workflow keeps a small local record of what the far end is believed to hold. Both that record and a batch’s own header already refused to proceed when they would not parse, and that part was right: treating a corrupt record as “the far end holds nothing” would send a full batch where an incremental was intended, which is worse than stopping.
What was wrong was the answer given. Neither refusal set a typed error, so both fell through to a generic catch-all whose category said the problem was environmental and whose destination state said unknown. Both statements were false. A record that will not parse is a data-integrity problem, not a host problem; and in both cases the engine knows exactly what has happened to the destination at the moment it refuses, which is nothing. The write side refuses before the batch is built at all. The read side refuses while reading the header, before the destination is touched.
Two codes were added, one per call site rather than one shared between them — the two failures happen at different points of different workflows, and a technician reading a code off a form needs to know which — each carrying the destination state untouched.
Then making that word true turned up something else. The apply path wrote a delivery receipt unconditionally, including on a failure where nothing had been validated. For an unreadable header the entry list is empty by construction, so it wrote a receipt for zero files — which created the destination directory if it did not already exist and put a file in it. That contradicts exactly what “untouched” promises. It was found by writing the test rather than by reading the code: the first draft asserted the destination was untouched and failed immediately, against a real directory holding a real, empty receipt.