I'm using Telerik.Windows.Controls.GridView for WPF, 2015.1.401.40, and had similar problems:
1 - ListView didn'd showed vertical scroll bar - setting MaxHeight to 400 brought it back
2 - but after that tracking down the scroll bar thumb didn't made the items list scroll.
Finally found the cause: Some time ago I had copied the standard style for ScrollViewer - provided by Microsoft (https://msdn.microsoft.com/en-us/library/cc278065(v=vs.95).aspx) - into my app.xaml. It was just for testing, I didn't made any changes to that!
After removing this style from the app.xaml the filter dialog worked fine.
This thread is a little bit old, but may be this helps someone...