Vague Explanations of Their Processes
If an SEO agency is not prepared to give you an explanation of how they operate, then the question that needs asking is ‘What have they got to hide?’.
With a poor agency, the answer is going to be either they do not have any processes and thus the campaign will be a shambles, or they use black hat techniques and do not want to reveal that to you.
Little or Poor Reporting
As part of your research, you should be asking what the reporting from the SEO agency is going to be. Good SEO agencies will have detailed and regular reporting, which is what you want to see.
However, poor agencies who know they are not going to get results will shy away from giving you regular reports, and if that is the case with any prospective agency you are looking at, do not even consider hiring them.