Abacus.AI at NeurIPS 2023

The Abacus.AI team has published five papers to appear at the Conference on Neural Information Processing Systems (NeurIPS) 2023. Three papers are in the main conference (including one awarded with an Oral Presentation) and two papers are in workshops (including one Invited Contributed talk). NeurIPS is a top machine learning conference held every December, showcasing the latest advancements in artificial intelligence. 

As evinced by the strong publications at NeurIPS this year, the Abacus.AI research team continues to produce top-quality, open source research. This year, the team has published foundational research on LLMs, tabular data, bias mitigation, and time-series forecasting.

We give brief descriptions of the NeurIPS 2023 papers below:

ForecastPFN: Synthetically-Trained Zero-Shot Forecasting

Many customers find that traditional time-series forecasting methods are inadequate for their applications because they do not have large training datasets. In many real-life scenarios, only a small number of initial observations, sometimes as few as 40, are available, restricting the applicability of standard forecasting approaches. While recent efforts have explored ‘zero-shot’ forecasting with limited initial data, their performance varies depending on pretraining data. In contrast, this study introduces ForecastPFN, the first zero-shot forecasting model trained exclusively on a novel synthetic data distribution. ForecastPFN, a prior-data fitted network, enables accurate and rapid predictions for new time series datasets in a single pass. Extensive experiments demonstrate that ForecastPFN outperforms state-of-the-art forecasting methods in terms of accuracy and speed, even when those methods are trained on hundreds of additional in-distribution data points.

Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition

Traditionally, model biases have been mitigated with approaches that focus on pre-processing, training penalties, or post-processing predictions. However, our research challenges this assumption, revealing biases inherent in neural network architectures. In this award-winning work, we conduct the first-ever neural architecture search for fairness alongside hyperparameter optimization. The resulting models not only outperform high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness on widely-used facial recognition datasets (CelebA and VGGFace2) but also demonstrate robust generalization across datasets and sensitive attributes. This work holds promise for customers seeking fair and accurate solutions in safety-critical domains.

Paper: https://arxiv.org/abs/2210.09943
NeurIPS Info: Talk on Dec. 14 at 10:15am in Ballroom A-C; Poster at 10:45am

When Do Neural Nets Outperform Boosted Trees on Tabular Data?

Tabular data holds a prominent role in machine learning, and the ongoing debate surrounding the superiority of neural nets (NNs) or gradient-boosted decision trees (GBDTs) for tabular data remains a topic of discussion. In this study, we shift the focus from the ‘NN vs. GBDT’ debate and conduct an extensive analysis across 176 datasets and 19 algorithms. Surprisingly, the performance difference between GBDTs and NNs is often negligible. We emphasize that light hyperparameter tuning on a GBDT can be more crucial than choosing between the two. Furthermore, we identify dataset properties that influence the suitability of NNs or GBDTs, providing practical insights for practitioners. To expedite tabular data research, we introduce the TabZilla Benchmark Suite, featuring 36 challenging datasets. This research aims to assist businesses in navigating the nuances of choosing the right approach for their specific tabular data needs.

Paper: https://arxiv.org/abs/2305.02997
NeurIPS Info: Poster on Dec. 14 at 5pm

Fairer and More Accurate Models Through NAS

Improving fairness in tabular data models has been a longstanding focus, usually involving adjustments to neural models after undesired outcomes. Our approach is distinct: we advocate for updating the model’s architecture and training parameters from the outset of the debiasing process, resulting in an entirely new model with superior performance. Our findings demonstrate that the proposed approach consistently outperforms state-of-the-art bias mitigation methods in fairness, accuracy, or both, while remaining Pareto-optimal over hyperparameters achieved through single-objective (accuracy) optimization runs. This research highlights the potential of automating fairness and accuracy optimization in deep learning models, providing businesses with a powerful tool to enhance the performance and fairness of their tabular data models.

Paper: https://arxiv.org/abs/2310.10628
NeurIPS Info: Dec. 15 in the Algorithmic Fairness through the Lens of Time

A Natural Experiment on LLM Data Contamination in Code Generation

For businesses considering the adoption of LLMs, it’s essential to be aware of potential issues related to data contamination—where evaluations might inadvertently include examples from the model’s extensive internet-based training data. Despite the complexity of measuring and addressing contamination concerns, this study pioneers a thorough examination using GPT models’ training cutoffs. By analyzing benchmarks over time, particularly focusing on datasets related to code and mathematical problem-solving, the research identifies significant trends that suggest potential data contamination. The findings underscore the importance for small businesses to carefully evaluate LLMs and consider best practices in benchmarking, especially given these models’ training on large-scale internet data. We provide an open-source dataset, raw results, and an evaluation framework, facilitating businesses in making informed decisions about the adoption of LLMs while considering potential data-related challenges. 

Paper: https://arxiv.org/abs/2310.10628
NeurIPS Info: Dec. 16 at 2:30pm in ICBINB Workshop

If you’re interested in learning more, you can come to our poster sessions and talks at the NeurIPS Conference, Dec. 11 – Dec. 16 in New Orleans, Louisiana, USA, or virtually. Visit https://neurips.cc/ to register.