Learning To Rank (LTR)

A curated list of research papers, software, datasets, and resources for Learning to Rank (LTR), also known as machine-learned ranking.

What is Learning to Rank?

Learning to Rank (LTR) is a machine learning approach for constructing ranking models from training data. LTR has been particularly successful in information retrieval, web search, and recommendation systems where the goal is to rank a set of items according to their relevance to a given query.

LTR Approaches

LTR algorithms can be categorized into three main approaches:

• Pointwise Approach: Treats ranking as a regression or classification problem, predicting a relevance score for each document independently.

  • Examples: McRank, Prank, OC SVM

• Pairwise Approach: Focuses on learning the relative order between pairs of documents. The goal is to minimize the number of incorrectly ordered pairs.

  • Examples: RankNet, RankBoost, RankSVM, LambdaRank, LambdaMART

• Listwise Approach: Directly optimizes ranking metrics (e.g., NDCG, MAP) by considering the entire list of documents.

  • Examples: ListNet, ListMLE, AdaRank, SoftRank, LambdaMART

Table of Contents

• Papers
  • 2003-2020
  • 2021-2025
• Software
• Dataset
• Others

Papers

2003

• Freund, Yoav, et al. "An efficient boosting algorithm for combining preferences." Journal of machine learning research 4.Nov (2003): 933-969.

2005

• Burges, Chris, et al. "Learning to rank using gradient descent." Proceedings of the 22nd international conference on Machine learning. 2005.

2007

• Xu, Jun, and Hang Li. "Adarank: a boosting algorithm for information retrieval." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.
• Yue, Yisong, et al. "A support vector method for optimizing average precision." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.
• Geng, Xiubo, et al. "Feature selection for ranking." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.
• Tsai, Ming-Feng, et al. "FRank: a ranking method with fidelity loss." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.
• Cao, Zhe, et al. "Learning to rank: from pairwise approach to listwise approach." Proceedings of the 24th international conference on Machine learning. 2007.
• Burges, Christopher J., Robert Ragno, and Quoc V. Le. "Learning to rank with nonsmooth cost functions." Advances in neural information processing systems. 2007.
• Zheng, Zhaohui, et al. "A regression framework for learning ranking functions using relative relevance judgments." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.
• Qin, Tao, et al. "Ranking with multiple hyperplanes." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 2007.

2008

• Amini, Massih Reza, Tuong Vinh Truong, and Cyril Goutte. "A boosting algorithm for learning bipartite ranking functions with partially labeled data." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Xu, Jun, et al. "Directly optimizing evaluation measures in learning to rank." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Veloso, Adriano A., et al. "Learning to rank at query-time using association rules." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Duh, Kevin, and Katrin Kirchhoff. "Learning to rank with partially-labeled data." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Guiver, John, and Edward Snelson. "Learning to rank with softrank and gaussian processes." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Zhou, Ke, et al. "Learning to rank with ties." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.
• Geng, Xiubo, et al. "Query dependent ranking using k-nearest neighbor." Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 2008.

2009

• Lease, Matthew. "An improved markov random field model for supporting verbose queries." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.
• Aslam, Javed A., et al. "Document selection methodologies for efficient and effective learning-to-rank." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.
• Donmez, Pinar, Krysta M. Svore, and Christopher JC Burges. "On the local optimality of LambdaRank." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.
• Cummins, Ronan, and Colm O'Riordan. "Learning in a pairwise term-term proximity framework for information retrieval." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.
• Banerjee, Somnath, Soumen Chakrabarti, and Ganesh Ramakrishnan. "Learning to rank for quantity consensus queries." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.
• Sun, Zhengya, et al. "Robust sparse rank learning for non-smooth ranking measures." Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 2009.

2010

• Burges, Christopher JC. "From ranknet to lambdarank to lambdamart: An overview." Learning 11.23-581 (2010): 81.
• Svore, Krysta M., Pallika H. Kanani, and Nazan Khan. "How good is a span of terms? Exploiting proximity to improve web retrieval." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 2010.
• Wang, Lidan, Jimmy Lin, and Donald Metzler. "Learning to efficiently rank." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 2010.
• Gao, Wei, et al. "Learning to rank only using training data from related domain." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 2010.
• Bagherjeiran, Abraham, Andrew O. Hatch, and Adwait Ratnaparkhi. "Ranking for the conversion funnel." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 2010.

2011

• Wang, Lidan, Jimmy Lin, and Donald Metzler. "A cascade ranking model for efficient ranked retrieval." Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 2011.
• Dai, Na, Milad Shokouhi, and Brian D. Davison. "Learning to rank for freshness and relevance." Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 2011.
• Ganjisaffar, Yasser, Rich Caruana, and Cristina Videira Lopes. "Bagging gradient-boosted trees for high precision, low variance ranking models." Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 2011.
• Cai, Peng, et al. "Relevant knowledge helps in choosing right teacher: active query selection for ranking adaptation." Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 2011.
• Chapelle, Olivier, and Yi Chang. "Yahoo! learning to rank challenge overview." Proceedings of the learning to rank challenge. 2011.

2012

• Wang, Lidan, Paul N. Bennett, and Kevyn Collins-Thompson. "Robust ranking models via risk-sensitive optimization." Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 2012.
• Severyn, Aliaksei, and Alessandro Moschitti. "Structural relationships for large-scale learning of answer re-ranking." Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 2012.
• Niu, Shuzi, et al. "Top-k learning to rank: labeling, ranking and evaluation." Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 2012.

2013

• Paik, Jiaul H. "A novel TF-IDF weighting scheme for effective ranking." Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 2013.
• Wang, Hongning, et al. "Personalized ranking model adaptation for web search." Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 2013.
• Raiber, Fiana, and Oren Kurland. "Ranking document clusters using markov random fields." Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 2013.

2015

• Grbovic, Mihajlo, et al. "Context-and content-aware embeddings for query rewriting in sponsored search." Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 2015.
• Severyn, Aliaksei, and Alessandro Moschitti. "Learning to rank short text pairs with convolutional deep neural networks." Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 2015.
• Vulić, Ivan, and Marie-Francine Moens. "Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings." Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 2015.

2016

• Ustinovskiy, Yury, et al. "An optimization framework for remapping and reweighting noisy relevance labels." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016.
• de Sá, Clebson CA, et al. "Generalized BROOF-L2R: A general framework for learning to rank based on boosting and random forests." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016.
• Wang, Xuanhui, et al. "Learning to rank with selection bias in personal search." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016.

2017

• Ibrahim, Muhammad, and Mark Carman. "Comparing pointwise and listwise objective functions for random-forest-based learning-to-rank." ACM Transactions on Information Systems (TOIS) 34.4 (2016): 1-38.
• Chen, Ruey-Cheng, et al. "Efficient cost-aware cascade ranking in multi-stage retrieval." Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017.
• Xiong, Chenyan, et al. "End-to-end neural ad-hoc ranking with kernel pooling." Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval. 2017. slide
• Su, Yuxin, Irwin King, and Michael Lyu. "Learning to rank using localized geometric mean metrics." Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017. slide
• Dehghani, Mostafa, et al. "Neural ranking models with weak supervision." Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017. slide
• Karmaker Santu, Shubhra Kanti, Parikshit Sondhi, and ChengXiang Zhai. "On application of learning to rank for e-commerce search." Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017.

2018

• He, Xiangnan, et al. "Adversarial personalized ranking for recommendation." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018. code
• Wang, Huazheng, et al. "Efficient exploration of gradient space for online learning to rank." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.
• Dato, Domenico, et al. "Fast ranking with additive ensembles of oblivious and non-oblivious regression trees." ACM Transactions on Information Systems (TOIS) 35.2 (2016): 1-31. slide
• Feng, Yue, et al. "From greedy selection to exploratory decision-making: Diverse ranking with policy-value networks." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.
• Ai, Qingyao, et al. "Learning a deep listwise context model for ranking refinement." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018. code
• Fan, Yixing, et al. "Modeling diverse relevance patterns in ad-hoc retrieval." The 41st international ACM SIGIR conference on research & development in information retrieval. 2018. code
• Lucchese, Claudio, et al. "Selective gradient boosting for effective learning to rank." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.
• Wu, Liang, et al. "Turning clicks into purchases: Revenue optimization for product search in e-commerce." The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.

2019

• Pasumarthi, Rama Kumar, et al. "Tf-ranking: Scalable tensorflow library for learning-to-rank." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.

2020

• Khattab, Omar, and Matei Zaharia. "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT." Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020. code
• Karpukhin, Vladimir, et al. "Dense Passage Retrieval for Open-Domain Question Answering." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. code
• Qu, Chen, et al. "Contextual Re-Ranking with Behavior Aware Transformers." Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.
• MacAvaney, Sean, et al. "Efficient Document Re-Ranking for Transformers by Precomputing Term Representations." The 43st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2020. code
• Zhuang, Honglei, et al. "Feature transformation for neural ranking models." Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.
• Lucchese, Claudio, et al. "Query-level Early Exit for Additive Learning-to-Rank Ensembles." The 43st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2020.
• Bevendorff, Maik Fröbe1 Janek, et al. "Sampling Bias Due to Near-Duplicates in Learning to Rank." The 43st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2020. code

2021

• Qin, Zhen, et al. "Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?" International Conference on Learning Representations (ICLR). 2021.
• Formal, Thibault, Benjamin Piwowarski, and Stéphane Clinchant. "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021. code
• Swezey, Robin, et al. "PiRank: Scalable Learning To Rank via Differentiable Sorting." Advances in Neural Information Processing Systems (NeurIPS). 2021.
• Oosterhuis, Harrie. "Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021. (Best Paper Award)
• Hofstätter, Sebastian, et al. "Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.
• Gao, Luyu, Zhuyun Dai, and Jamie Callan. "Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline." European Conference on Information Retrieval (ECIR). 2021.
• MacAvaney, Sean, Franco Maria Nardini, and Raffaele Perego. "A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.
• Formal, Thibault, et al. "A White Box Analysis of ColBERT." European Conference on Information Retrieval (ECIR). 2021.

2022

• Santhanam, Keshav, et al. "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction." Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2022. code
• Formal, Thibault, et al. "From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022. code
• Pobrotyn, Przemysław, et al. "Learning Neural Ranking Models Online from Implicit User Feedback." Proceedings of the ACM Web Conference 2022. 2022.
• Khosla, Sopan, and Vinay Setty. "Risk-Sensitive Deep Neural Learning to Rank." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.
• Pasumarthi, Rama Kumar, et al. "Learning-to-Rank at the Speed of Sampling." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.
• Hofstätter, Sebastian, et al. "Ensemble Distillation for BERT-Based Ranking Models." Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR). 2021.
• Lassance, Carlos, et al. "Learned Token Pruning in Contextualized Late Interaction over BERT (ColBERT)." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.
• Yang, Tao, et al. "Can clicks be both labels and features? Unbiased Behavior Feature Collection and Uncertainty-aware Learning to Rank." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.

2023

• Buyl, Maarten, et al. "RankFormer: Listwise Learning-to-Rank Using Listwide Labels." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.
• Ai, Qingyao, Xuanhui Wang, and Michael Bendersky. "Metric-agnostic Ranking Optimization." Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.
• Zhao, Haiyuan, et al. "Unbiased Top-k Learning to Rank with Causal Likelihood Decomposition." Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP). 2023.

2024

• Khramtsova, Ekaterina, et al. "Leveraging LLMs for Unsupervised Dense Retriever Ranking." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024. (Best Paper Award)
• Borisyuk, Fedor, et al. "LiRank: Industrial Large Scale Ranking Models at LinkedIn." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.
• Huang, Xuyang, et al. "Unbiased Learning-to-Rank Needs Unconfounded Propensity Estimation." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.
• Jagerman, Rolf, et al. "Unbiased Learning to Rank: On Recent Advances and Practical Applications." Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.
• Liu, Yu-An, et al. "Multi-granular Adversarial Attacks against Black-box Neural Ranking Models." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

2025

• Wang, Qingyu, et al. "From Features to Transformers: Redefining Ranking for Scalable Impact." arXiv preprint arXiv:2502.03417. 2025.

Software

Classical LTR Libraries

• Support Vector Machine for Ranking - SVM^rank implementation
• Support Vector Machine for Optimizing Mean Average Precision - SVM-MAP implementation
• jforests - Java implementation of gradient boosted trees for LTR
• ListNet - Listwise approach implementation
• ListMLE - List Maximum Likelihood Estimation
• Metric Learning to Rank - Python implementation
• Lerot - Online learning to rank framework
• xapian-letor - LTR module for Xapian search engine

Deep Learning & Modern LTR Libraries

• TensorFlow Ranking - Scalable TensorFlow library for LTR
• LambdaRank Example on LightGBM - LambdaRank with gradient boosting
• Chainer implementation of RankNet - Neural network approach
• OpenNIR - Neural IR research platform
• metarank - Modern LTR for e-commerce and recommendations
• Transformer Rankers - Library for ranking experiments with transformers

Dataset

• LETOR 3.0/4.0 - Benchmark datasets for learning to rank research
• MSLR WEB10K/WEB30K - Microsoft Learning to Rank datasets
• TREC QA Track Data - Question answering and retrieval datasets
• Yahoo! Learning to Rank Challenge - Large-scale LTR dataset from Yahoo!

Others

Tools and Projects

• QuickRank - Fast learning to rank C++ library
• ExpediaLearningToRank - LTR application for hotel search

Japanese Resources (日本語リソース)

• ランク学習（Learning to Rank） Advent Calendar 2018
• DSIRNLP#1 ランキング学習ことはじめ
• Learning to rank (LTR) とは何か
• SIGIR2011読み会 3: Learning to Rank
• SIGIR2012勉強会 23: Learning to Rank

Contributing

Contributions are welcome! If you know of any important papers, software, datasets, or resources related to Learning to Rank that are not listed here, please feel free to:

1. Open an issue with the details
2. Submit a pull request with your additions

When adding papers, please:

• Include the full citation with authors, title, and venue
• Add a link to the paper (preferably direct PDF or DOI)
• Place the paper in the appropriate year section
• Follow the existing format

Star History

If you find this repository useful, please consider giving it a star ⭐

License

This list is provided as-is for educational and research purposes. All linked papers and resources are copyright of their respective authors and publishers.