In a saucepan, over medium-high heat, bring the plums, blackberries, blueberries, half the raspberries, the sugar, and ¼ cup water to a simmer. Cook, stirring occasionally, until the blueberries begin to burst, about 5 minutes.
Remove from heat. Stir in the lemon zest and juice and the remaining raspberries. Transfer to a bowl and let cool for 15 minutes.
Line an 8-inch square baking dish with 2 sheets of plastic wrap, allowing at least 6 inches to hang over each side. Place the bottoms of the buns in the dish, cut-side up. Spoon half the fruit mixture over the buns.
Top with the remaining buns, cut-side up, and the remaining fruit mixture. Cover with the overhanging plastic wrap, a plate, and two 15-ounce cans. Refrigerate until chilled, at least 1½ hours. Spoon the pudding into bowls and top with whipped cream (if using).