Wondering if you need to take an umbrella before stepping outside? The weather forecast can guide you, but only if it’s accurate.
Predicting things like the weather or air pollution involves using known data from certain areas to make educated guesses about unknown ones. While scientists often rely on established methods to validate these predictions, researchers at MIT have found that these common methods don’t always work well for spatial predictions.
The MIT team discovered that traditional validation methods can lead to false confidence in forecast accuracy. They created a new approach that assesses these methods more effectively and found that two classic validation techniques can be misleading in spatial contexts. Their goal was to understand why these techniques fail and develop a more reliable alternative.
Their new method showed better results in experiments with real and simulated data. For example, they successfully predicted wind speeds at Chicago O’Hare Airport and forecasted temperatures in five major U.S. cities. This method has broader applications too, from helping climate scientists estimate sea temperatures to supporting public health experts in assessing the impact of air pollution on health.
“We hope this will lead to more reliable evaluations of new predictive methods and a clearer understanding of their performance,” says Tamara Broderick, an MIT associate professor involved in this research. Her team included lead author David R. Burt and graduate student Yunyi Shen, and their findings will be presented at an upcoming conference.
Broderick’s group, in collaboration with oceanographers and atmospheric scientists, aimed to improve machine-learning prediction models for spatially-related problems. They noticed that traditional validation methods often fall short when dealing with spatial data. These methods typically keep a small portion of training data aside for validation, but they rely on assumptions that don’t hold true for geographic contexts.
For instance, when validating a model predicting air pollution using EPA sensor data, the sensors aren’t truly independent. Their positions are influenced by other sensor locations, which can skew results. Additionally, data from urban sensors differ significantly from that in rural areas. These discrepancies can lead to misleading outcomes when the assumptions of independence and identical distribution are applied.
To tackle this issue, the researchers proposed a new assumption tailored specifically for spatial contexts, where data from nearby locations tends to vary smoothly. For instance, pollution levels usually don’t change drastically from one house to the next. This assumption provides a more accurate ground for evaluating spatial predictors.
Using their technique is straightforward: input the predictor, specify the locations of interest, and provide validation data. The method then evaluates how reliable the predictions are for those areas. However, finding ways to assess this innovative validation method was challenging. Broderick explains, “We had to rethink how we evaluate an evaluation.”
Initially, they conducted tests with simulated data to control variables, followed by modifying real data to create semi-simulated datasets. Finally, they validated their approach with actual data. By testing their method against three types of realistic scenarios—like predicting property prices based on location and forecasting wind speeds—they found that their technique often outperformed traditional methods.
Looking ahead, the researchers aim to refine uncertainty quantification in spatial predictions and explore other fields where their method could enhance predictive performance, including time-series data.
This research received support from the National Science Foundation and the Office of Naval Research.
Source link
Tamara Broderick, Spatial prediction methods, Validation methods
