what gives red velvet cake its flavor? the distinct flavors of red velvet cake come from the buttermilk, vinegar, cocoa powder, and cream cheese frosting. some red velvet cake recipes use only a tablespoon or two of cocoa powder, giving virtually no chocolate flavor.