Thanks for all the good ideas! I found a vaccum leak at the connection under the throttle body (not completely sure what it is for). Was definitly the source of the problem, because it doesn't go way up in RPMs now when the clutch is in. It still takes a little longer to go down in RPM than I think it should, so i'm thinking there is still a small vaccum leak...Most of the hoses on the intake have seen better days, so i'm thinking they're not sealing as well as they should. I'm going to look at the fuel trim data on my scanner to see if it indicates any more vaccum leaks...
The throttle body I installed is the bolt in kind, and the gasket for the top plate has seen better days...I'm pretty sure its not leaking, but definitly something I should replace soon. I love fixing dumb stuff like this

haha