Spread the cake with jam. Cut into one-inch cubes and arrange them in a large, decorative bowl. Pour the sherry over cake and cover with the peaches. In a saucepan, combine the milk and cornstarch. Bring to boil, stirring constantly. Add the vanilla and sugar; stir until thickened (like pudding). Remove from heat, and let cool. Pour the cooled pudding over cake in bowl. Sprinkle the top with the almonds.
Spread the whipped topping over the almonds and top with the berries.