Toss the peaches, cinnamon, nutmeg, and sugar in a mixing bowl to coat evenly. Set aside.
Preheat the oven to 350F.
Combine the butter, sugar, baking powder, baking soda, salt and eggs in a mixing bowl. Blend the mixture, working it with the back of wooden spoon to break up any lumps, until it is smooth. Add the vanilla, cinnamon, nutmeg, and milk. Add the flour and beat until smooth.
In a 10 1/2-inch ovenproof skillet (or an 8 x 11 1/2 x 2-inch baking pan), combine the butter and the sugar over medium-high heat. Stir with a wooden spoon until the mixture caramelizes and becomes syrupy.
Spread the peaches evenly in the bottom of the skillet or pan. Remove from the heat. Pour the cake batter into the skillet or pan and spread evenly over the peaches.
Bake for about 40 minutes, or until the cake is golden and pulls away slightly from the sides of the skillet or pan. Remove from the oven and cool for about 30 minutes, then invert over a large platter.
Slice to serve.
Yield: One 8- to 9-inch cake, 12 servings