we would not recommend adding extra almonds as the cake could become too greasy as the almonds will release some oil during cooking. some polenta can be quite coarse so to try and reduce the "gritty feel" you could try using fine ground cornmeal, though it will not be eliminated completely.