To find potent and selective inhibitors of dipeptidyl peptidase IV (DPP-IV), we synthesized a series of 2-cyanopyrrolidine with P2-site 4-substituted glutamic acid derivatives and tested their activities against DPP-IV, DPP8, and DPP-II. Analogues that incorporated a bulky substituent at the first carbon position of benzylamine or isoquinoline showed over 30-fold selectivity for DPP-IV over both DPP8 and DPP-II. From structure-activity relationship studies, we speculate that the S2 site of DPP8 might be similar to that of DPP-IV, while DPP-IV inhibitor with N-substituted glycine in the P2 site and/or with a moiety involving in hydrophobic interaction with the side chain of Phe357 might provide a better selectivity for DPP-IV over DPP8.