Whisk together whole milk with sugar, cornstarch, and salt. Slowly bring to a simmer over medium heat, stirring frequently.
Whisk the egg yolks and slowly pour in the milk mixture, whisking constantly. Make sure not to add the milk too fast or the eggs will scramble. Bring the mixture back to the stove and heat until thickened over low heat whisking constantly, about 4-5 minutes.
Remove from heat and stir in the cream cheese and vanilla until the cream cheese has melted. Transfer to a clean bowl and cover with plastic wrap, letting the plastic wrap directly touch the pudding. Let cool.
Whip the heavy cream until stiff peaks form. Add the powdered sugar and whip until combined.
Assemble the pudding. Spread half of the pudding in the bottom of a 9x13-inch baking dish or into individual cups. Lay half of the graham crackers on top in an even layer followed by half of the sliced banana. Spread the remaining pudding followed by the remaining graham crackers and bananas. Finish with whipped cream. Let chill for at least 4 hours or overnight. Before serving, sprinkle crumbled graham crackers on top if desired.
Notes
You can make the pudding ahead of time and assemble the pudding parfaits the following day, if desired.