Search speed slow using 'All Fields Search' vs 'Search Panel'

If I'm using the search directly from the list page (top) the searches take way too long for some reason. If I use the the search panel on the left side, they are almost instant. I was wondering if anyone has any suggestions on why it may take longer using one over the other?
There are about 4,000 records with 20 fields.
To narrow down the issue, I only selected 2 search fields for each type of search in the "Search and Filter settings..."
For example, I selected "ClientID" and "ClientName" for 'Search panel' and 'Include into All Fields Search' columns.
The search bar on the left shows both fields. If I use one or both fields during my search, the results are instant.
If I search using the search bar at the top of the page, it may take over 15 seconds.
I almost feel like the top bar is searching every single field even though it's only returning the results from the selective columns.
I have 'highlight search results' and 'show search suggest' enabled. No other options enabled.
Has anyone come across this before and what can I do to improve the search speed of the top search bar?