First up if you haven’t been spraying with cooled boiled water this can cause other things to grow, eg: Untreated tank water.
Being exposed to the elements in your growing area in between flushes while the block is resting in a weakened state can allow foreign elements to start growing.