Generally, the z-index values are determined mostly by requirements related to which widget should be on top of other widgets. For example, selection marquees and elements related to dragging should be on top of everything else. Tooltips should be over Windows, except validation tooltips. There is no way to handle (even predict) all possible integration / overlay scenarios perfectly, so we just use common sense and amend the z-index values when necessary. There are no fixed rules.
It is also worth mentioning that some z-index values are calculated on the fly, depending on z-index values applied to affecting elements on the page. For example this is how DateTimePickers work to ensure that their popups open on top of everything else.
Join us on our journey to create the world's most complete HTML 5 UI Framework - download Kendo UI