You don’t actually have to attach the texture to a material before using it in a displace modifier. In the Textures tab, you will see 3 icons for selecting what type of texture you want to set: Material, World or Brush. Pick Brush, set up your texture there, then attach it in the displace modifier.