the main difference is the texture, which is determined by two things: the size of the sugar crystals in the candy, and the concentration of the sugar. toffee is smooth with no sugar crystals, whereas fudge has tiny crystals that give it that texture.