Stir the cocoa powder into the marshmallow fluff in a medium bowl, adding a little at a time and stirring until smooth between additions. Spread evenly among 6 of the graham cracker squares. Divide the ice cream among the remaining squares, spreading evenly. Sandwich the squares together and freeze on a wax paper–lined pan until the ice cream is firm, at least 4 hours and up to overnight.
Microwave the chocolate in a microwave-safe bowl on high in 20-second intervals, whisking in between, until melted and smooth. Add the oil and stir to combine. Dip half of each sandwich in the melted chocolate and return to the pan. Freeze until the chocolate sets, about 10 minutes. Keep frozen, covered, for up to 3 days.