I shaved with oil a few years back (shave oil, not olive oil) with carts not DE.
I found that even after washing there was a thin film of oil on my face all day, and my pillow ended up getting pretty dirty from sleeping with an oily face.
I haven't tried since, maybe I could've washed my face better - the soaps and creams available are nicer than oil anyway, why bother...