Heat a small skillet over medium heat. Toast fennel seeds in the dry skillet until fragrant and beginning to brown. Remove to a large bowl.
Add all remaining ingredients, except the buns, to the bowl and combine with your hands until well incorporated. Divide into 4 large or 6 smaller patties and let chill at least 15 minutes.
Heat an outdoor grill or grill pan over medium heat and grill 6 minutes a side or until firm. Serve on toasted buns with mayonnaise and cole slaw.