I have a BT Elm 327 OBD II connector, and I haven't noticed any significant battery drain, and I leave mine in 24/7, and I don't usually drive that vehicle on the weekends.
I would guess that maybe your battery is weak, because I can't see a bluetooth connector pulling enough power to drain a good battery.

It's quite difficult to diagnose this kind of problem as it could quite easily be the battery or ECU type dependant - many "failing" car batteries would be able to retain their charge for a few days but even a constant 30mA draw could flatten them in that time, so you could drive your car, but leave it more than a week and you would be jump starting it or notice that it would 'sound like it was more effort to start'.

The Adapter could also be keeping the ECU in a high power mode - this would be vehicle specific however. Most ELM bluetooth adapters revert to a low power state when not connected (to the phone) so should not flatten the battery. I know that later ELM chipsets (versions 1.4 and above) now have support for a proper 'sleep' command, but this should not even be needed to be used in most cases as they go to sleep in previous versions quite happily.

I would hazard a guess that you have a battery on the way out and should get it tested before considering replacing it, or the ECU is being kept in an alive state by the adapter being present on the OBD diagnostic databus and therefore consuming energy... or you just left your lights on and didn't notice 😉