The TeamPulse installer is built using WiX (Windows Installer XML), a free and open source toolset originally created by Microsoft. More information can be found on the official WiX Toolset page, or via the WiX CodePlex page.
Any professional installer package is definitely a challenging experience, and working with WiX is no different. However it is very flexible, and there is a fair amount of information available on the web for consumption.
Best of luck on your installer!
the Telerik team