Preheat your oven to 350°F (175°C) and line a muffin tin with cupcake liners.
In a small saucepan, heat the milk until it just begins to simmer. Remove from heat, add the Earl Grey tea bags, and let steep for 10 minutes. Remove the tea bags and let the milk cool to room temperature.
In a mixing bowl, cream together the butter and sugar until light and fluffy. Add the eggs one at a time, beating well after each addition. Stir in the vanilla extract.
In another bowl, whisk together the flour, baking powder, and salt. Gradually add the dry ingredients to the butter mixture, alternating with the cooled tea-infused milk, beginning and ending with the dry ingredients. Mix until just combined.
Divide the batter evenly among the cupcake liners, filling each about two-thirds full. Bake for 18-20 minutes, or until a toothpick inserted into the center of a cupcake comes out clean. Allow the cupcakes to cool in the tin for 5 minutes, then transfer to a wire rack to cool completely.