Using hotkeys can make this process faster. Click a gem from the panel, hold CTRL, and click the lowest (tier 1) row in your inventory. It will flood the inventory entirely.
To get rid of a gem quickly, you can hover your mouse over it and press X on your keyboard to sell it.