Heat oven to 375° F. Place the halibut in a 9-inch square baking dish.
Pour the wine over the fish, drizzle with 1 tablespoon of the oil, and season with ½ teaspoon salt and ¼ teaspoon pepper. Scatter the olives, rosemary, and garlic over top.
Bake until the fish is the same color throughout and flakes easily, about 15 minutes.
Meanwhile, heat the remaining oil in a skillet over medium heat. Add the potatoes and cook, stirring occasionally, until golden and tender, 15 to 17 minutes.
Transfer the potatoes to a paper towel-lined plate. Season with ½ teaspoon salt and ¼ teaspoon pepper. Serve with the halibut and olives.