Preheat oven to 350 degrees F.
Slice the apricots in half and each half into 3 wedges for a total of 6 wedges per apricot. Toss the apricots and blackberries with the brown sugar and cinnamon and set aside for 15 minutes.
Combine the dry ingredients for the cake including the flour, sugar, ground hazelnuts, baking powder, salt, cinnamon, and nutmeg.
Whisk together the vanilla, buttermilk, and egg until well combined. Add the wet ingredients to the dry, stirring just until combined.
Melt ¼ cup butter in a 10-inch cast iron pan over medium heat. Continue to heat the butter until it turns golden brown and smells nutty, about 3-4 minutes. Remove from heat and pour the cake batter into the center of the pan, letting the butter surround the batter.
Arrange the fruit on top of the cake and sprinkle the chopped hazelnuts on top. Bake the blackberry apricot buckle for 45 minutes - 1 hour or until a toothpick inserted in the center comes out clean. Let cool and dust with powdered sugar before serving.