I've got a fun one for you!
CustomUpdatePanel can be a control with no code, i.e. it doesn't change the behaviour of the UpdatePanel at all, and it still fails to work properly.
If a wrap the tree like so:
Then it works again.
From the behaviour, I'm guessing there's some code that says something like:
if (AncestorControl.GetType() == typeof(UpdatePanel))
if (AncestorControl is UpdatePanel)
Just a guess though!
I just found the ForceAjaxRenderer property, and I'm guessing that will fix my problem.