Preheat the oven to 350°F (176°C). Grease a half-sheet pan and line the bottom with parchment paper. Set aside.
In a bowl, combine the flour, baking powder, baking soda, salt, and spices. Whisk together and set aside.
Fit your stand mixer with a paddle attachment and beat the eggs and brown sugar until fluffy. Add the oil and pumpkin, and beat until well combined. Gradually mix in the dry ingredients in three batches, alternating with the milk. Do not overmix.
Pour the batter into the sheet pan and use an offset spatula to evenly spread it throughout the pan. (This batter tends to stay where you put it, so if it is spread unevenly, it will likely bake that way.)
Bake for 12 to 15 minutes, or until the cake is golden and cooked through. Do not overbake. Let cool. Place a sheet of parchment paper on the countertop and carefully flip the cake onto it. Peel off the parchment paper and use a 3-inch (7 cm) cutter to cut the cake into rounds. Break the remaining cake into crumbs using your hands. These crumbs will be used as the topping.
Whip the cream cheese and butter until light and fluffy. Add the vanilla and powdered sugar, and mix until well combined.
Place one cake round on a piece of parchment and spread 2 tablespoons of frosting on top. Repeat with 2 or 3 more layers. Top with cake crumbs and serve immediately.