The selection can be conducted in a reliable manner as follows:

First, screen all records identified in databases, hand searching, citation tracking and other search strategies by reading the titles and abstracts. To minimise bias, this should be done by two independent reviewers. Retrieve the full-text of all records that are included by both reviewers, and records for which reviewers disagree (no discussion needed as there is not enough information in the abstract). In other words, exclude only records that are excluded by the two reviewers.

Second, read the full-text of the included records. To minimise the selection bias, this step should be done by two independent reviewers. Disagreements between reviewers can be solved by discussion, and a third party when needed.

Regarding each step, an inter-rater agreement can be estimated using Cohen’s Kappa. 

A new tool has been developed to help reduce the number of records to screen: the Automated Text Classifier of Empirical Research (ATCER). The ATCER automatically categorizes publications indexed in bibliographic databases into (a) empirical studies (using qualitative, quantitative and mixed methods), and (b) non-empirical work (commentary, editorial, literature review, method paper, program description, and professional guideline, among other examples). If your review only include empirical studies, this tool can be useful to identify and remove non-empirical work.  This tool was developed from a study comparing the performance of different automated text classification methods (Langlois et al, 2018). More information and access to the ATCER are available at this website: https://atcer.iro.umontreal.ca/. 

Reference: 

• Khan, K., Kunz, R., Kleijnen, J., & Antes, G. (2011). Systematic Reviews to Support Evidence-Based Medicine: CRC Press.
• Langlois A, Nie JY, Thomas J, Hong QN, Pluye, P. (2018). Discriminating between empirical studies and nonempirical works using automated text classification. Research Synthesis Methods. DOI: 10.1002/jrsm.1317