You are not going crazy. We’ve had several incidence of nest usurpation and supersedures. This is where one species overtakes the nest of another species, sometimes completely ripping out a finished nest! Evil, I know, but this tells us that nesting sites are at a premium.
Here’s what happened in my block. On June 29th, I had a nicely completed nest of a “debris” wasp (Solierella sp.) in tunnel J3. When I was out photographing my block on July 7th, I noticed J3 looked empty, but after awhile, I saw a bee in the tunnel. Hmm. Looks like the Solierella nests in the column 4 holes are safe, but nests in the column 3 holes are vulnerable.