Place the sponge pieces and biscuits into the base of a deep glass dish.
Spoon 150ml of sherry and 1 tbsp of cognac over the base and let soak.
Warm the jam until runny and pour it over the soaked sponge and biscuits.
Layer the fresh berries and 85g of flaked almonds over the jam.
Pour 600ml of ready-made custard over the fruit layer.
Stir 125ml sherry, 2 tbsp brandy, lemon juice, and sugar together until dissolved.
Add the cream and nutmeg, then whisk until the mixture holds its shape.
Spoon the syllabub mixture over the trifle and chill, ideally overnight.
Garnish with orange zest and additional flaked almonds before serving.