stir almond flour, eggs, cayenne pepper, salt, and black pepper into cauliflower until well-combined; form into 1/4-cup sized patties. heat 1 teaspoon olive oil in a skillet over medium heat. fry patties in a single layer until golden brown, about 5 minutes per side. repeat with remaining patties and olive oil.