Review history

How Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP got from its first submission to CIKM 2024: 5 submissions. Scores use each venue's own scale (hover a score for the reviewer's confidence; 3→5 = changed after the rebuttal). ← CV of Failures

Aug 2023CIKM 2023mean 0.3, rejected
Dec 2023AAAI 2024rejected
Feb 2024TheWebConf 2024mean 0.4, rejected
May 2024KDD 2024mean 2.8, rejected
Jul 2024CIKM 2024mean 0.5, accepted

About the full reviews on this page. Reviews that are already public on OpenReview are only linked. For the other rounds I share the reviews my paper received myself, under three rules: only for papers that were eventually accepted, only conference reviews (never journal reviews), and only as received — reviewers appear under the anonymous IDs or numbers the venue gave them, and I do not know who they are. The venue rules I checked are quoted with each round; none of them prohibits authors from sharing the reviews of their own submission.

CIKM 2023 ✗ rejectedsubmitted 2023-05-27
-111mean 0.3
reviews not public — shared by the authors download as .md

What the reviewers raised

  • Applied Research track; overall evaluations -1, 1, 1; rejected.
  • The main criticism was the single Korean GDP dataset; reviewers also asked how the macroeconomic indicators were chosen, about publication delays of the indicators, and whether cubic-spline interpolation leaks future data.

What changed for the next submission

  • Added a second dataset: from the AAAI 2024 version on, the paper evaluates on GDP data from South Korea and the United Kingdom.

CIKM is an ACM conference and does not publish reviews. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”.

Full reviews
Reviewer 1: rating -1 (weak reject)

Overall evaluation -1 (weak reject)

Strengths

(1) The authors clearly describe three research questions they want to address, and conduct the experiments with them. The research questions are meaningful and common in the real-world application.
(2) The illustrations (diagrams, formula etc.) provided are comprehensible.
(3) The proposed method consistently outperforms the baseline across all evaluation tasks.

Weaknesses

(1) The task used for evaluation is somewhat singular. Its empirical study was barely conducted on macroeconomic data from South Korea. This method should be able to applied to other similar issues facing the same problems (as illustrated in the 3 research questions). However, this work did not show this, even did not show whether it could be applied to other regions. Thus, cannot convincingly show its great impact to the real-world.

(2) It lacks of description of the selection of those macroeconomic indicators. This is very important in economic research. In this study, I think the report time of those macroeconomic indicators are more critical than the time labels, as some macroeconomic indicators might also have delay in time of publication like the GDP. Nowcasting should guarantee only use those data accessible before the current time.

(3) 1 have doubts about the science and applicability of using cubic spline interpolation in the method. Which point, and how many points are involved for the cubic spline interpolation, should be justified. This fitting method is sensitive to the data points. If all data points in train/dev/test set are used for that, there might be an information leakage from the future time step. If only the data points from the previous time step are involved here, it might be better to provide some details on how to process the first few intervals. Because it makes little sense to fit a cubic spline to fewer than five points.

Overall evaluation (comments)

Based on the Weaknesses (1) and concerns in Weakness (2) and (3), I tend to weak reject this paper. I think this paper could be improved.

Reviewer 2: rating 1 (weak accept)

Overall evaluation 1 (weak accept)

Strengths

1. The problem studied in this paper is interesting.
2. The proposed method is reasonable.
3. The experimental results seem promising.

Weaknesses

1. It would be better if more datasets are used for experiments.

Overall evaluation (comments)

The paper introduces a new model called NCDENow, which combines Nonlinear Continuous Dynamic Equations (NCDEs) with Dynamic Factor Models (DFMs) to improve the accuracy of nowcasting GDP growth. The proposed method addresses challenges such as sudden drops and irregular or missing data more effectively. The researchers conducted experiments to evaluate the performance of NCDENow in different scenarios, considering factors like missing data rate and volatility of economic data. The results showed that NCDENow outperformed various versions of Recurrent Neural Networks (RNNs) and DFMs in nowcasting GDP.

Pros:

1. The problem studied in this paper is interesting.

2. The proposed method is reasonable.

3. The experimental results seem promising.

Cons:

1. It would be better if more datasets are used for experiments.

Reviewer 3: rating 1 (weak accept)

Overall evaluation 1 (weak accept)

Strengths

The model is well explained. The experimental setting is clear and the results are promising.

Weaknesses

some minor issues with the choice of the dataset and the lack of additional datasets to compare with.

Overall evaluation (comments)

In this paper, the authors introduce a novel framework for nowcasting GDP, making the following contributions:
- They adopt the framework of Nonlinear Continuous Dynamic Equations (NCDEs) to capture the dynamics of continuous macroeconomic indicators.
- They propose a method to address the dynamic factor loading and idiosyncratic disturbance, enabling accurate calculation of GDP growth. The experiments on GDP nowcasting are conducted on the Korean GDP dataset and compared with 6 baselines.

As far as my knowledge is concerned, the model seems well explained. The experimental setting is clear and the results are promising.

I think that there might be some minor issues with the choice of the dataset and the lack of additional datasets to compare with.

- Can you share the dataset you collected from Economics Statistics System? It would be important for both reproducibility and baseline studies.

- Independently from the country, could you add in your experiment a dataset already used by other papers, such as https:// onlinelibrary.wiley.com/doi/full/10.1002/jae.2845, to see how your approach works on datasets and methods optimized on that particular dataset?

- The improvements shown in Table 2 and Table 3 are very likely significant from a statistical significance point of view. What about the computational complexity of your algorithms compared to the baselines?

Meta-review and decision
Meta-review by senior PC member, 2026-10-01
Meta-review

The reviewers found this work interesting and have provided several concrete suggestions to further improve the paper. I encourage the authors to consider the detailed reviewer feedback and address them as appropriate.

AAAI 2024 ✗ rejectedsubmitted 2023-08-15
Weak RejectReject
reviews not public — shared by the authors download as .md

What the reviewers raised

  • Overall evaluations: weak reject, reject; not promoted to phase 2.
  • Both reviewers saw the method as a simple combination of DFMs and NCDEs, asked how NCDENow differs from the DFM-NCDEs baseline, and found the motivation and writing unclear.

AAAI-24 did not publish reviews. Its ethical guidelines for reviewers ask reviewers to keep the papers and their reviews confidential; nothing restricts authors from sharing the reviews of their own submission.

Full reviews
Reviewer 3: rating Weak Reject, confidence Quite confident

Reproducibility Yes, Overall evaluation Weak Reject, Confidence Quite confident

Summary

This paper proposes a method combining NCDEs with DFMs to nowcast GDP. According to experiments on real-life datasets, the model performs well on sudden drops and irregular or missing data.

Strengths and weaknesses

The paper is well written, and contributes a novel framework combining NCDEs and DFMs for nowcasting GDP, which enables better nowcasting of irregular time series to solve the real-world problems that irregular times and lagged factors often interfere with GDP nowcasting. And the proposed scheme outperforms the baselines and can be applied to predict various economic indicators to help make better decisions, which has significant practical value in real life.

However, there exist several problems, which must be solved before it is considered for publication.
- The abstract fails to state the key contents of the paper clearly. The proposed model is a fusion of NCDEs and DFMs. However, in the abstract, the author only focuses on NCDEs and mentions DFMs in one stroke, which may cause readers to misunderstand the content and methods of the paper or ignore one of the key parts. It is recommended that the author should explain more clearly the key features and uses of the DFMs to ensure that readers can obtain a comprehensive understanding when reading the abstract.
- The contribution of the article is not obvious enough, and the novelty of the paper seems limited that the fusion method simply stacks two models without an in-depth exploration of the complementarity and synergy between these models. The paper also doesn’t clearly and detailedly explain why combining NCEDs and DFMs helps better deal with lagged and missing data in theory.
- The experiment in this paper is not convincing enough. Some of the baselines compared to the proposed model are relatively out of date. And since the authors choose to use MLP as one of the methods to design the CDE function at last, it’s more reasonable and comprehensive to add baselines related to MLP models like DFM-MLP.
- The structure of the model is not clearly illustrated. Figures in the paper do not provide intuitive demonstrations of how important parts in the model work.
- The paper lacks an explanation of the advantages and disadvantages of the models mentioned in related works. The authors need to highlight the advantages of the proposed model compared to existing models.
- When elaborating the conclusions of section 4.6, NCDEs are missing from the ranking.

Questions for the authors

1.There is a model called DFM-NCDEs in the baselines of your experiment. What’s the difference between it and your proposed model as they both seem to be a fusion of DFM and NCDEs?
2.From your paper, it seems that your model simply combines two technical methods and is not innovative enough. Can you explain the novelty of your proposed framework?

Reviewer 5: rating Reject, confidence Somewhat confident

Reproducibility Yes, Overall evaluation Reject, Confidence Somewhat confident

Summary

This paper proposes a novel method for GDP nowcasting by combining NCDEs and DFMs. The key idea is to use DFMs for estimating latent factors and NCDEs for capturing temporal dependencies and idiosyncratic disturbances. Experiments on real world GDP datasets validate the effectiveness of the proposed method compared with other baselines. Extensive analysis on the experiment results were carried out thereafter.

Strengths and weaknesses

- Strength
(1) The paper leverages DFMs for latent factor estimation and NCDEs to capture temporal dependencies and idiosyncratic disturbances.
(2) Experiment results show great improvements on real world datasets.

- Novelty: poor
Since NCDEs and DFMs have already been successfully applied for time series nowcasting, a simple integration of NCDEs and DFMs may not be novel enough.

- Quality: fair
The proposed method makes sense: employing two models (NCDEs and DFMs) to capture model different aspects of the time series.

- Clarity: poor
(1) Writings: overall, the writing is quite poor making it hard to follow for general audience, including but limited to typos/notation confusion, vague motivation, poor explanation for the experiments, etc.
(2) Motivation for combining NCDE and DFM is not clear.
(3) notations and typos:
a. eq. (2) how can the vector \(\bm{\Lambda z_t}\) plus a scalar results in a scalar \(y_t\)? Similar situation applies to eq. (11).
b. please use bold character for matrices, e.g., \(X\) in eq.(1), \(A_i\) in the paragraph after eq.(3), etc.

- Significance: poor
Although the empirical experiment parts seem convincing, its significance is hampered by its poor novelty and clarity. More analysis and explanation regarding empirical performance are recommended to better justify the proposed method's significance.

Questions for the authors

More explanations are needed for experiment results:
(a) What is ‘DFM-NCDEs’ in table 1? What the difference between ‘DFM-NCDEs’ compared with the proposed ‘NCDENow’?
(b) Compared with ‘DFM-NCDEs’, what contributes most to the significant performance improvement of the proposed ‘NDCENow’?
(c) In table 1, it seems that the integration of DFM and NCDEs (i.e., DFM-NCDEs) performs poorer than the pure DFM and pure NCDEs. For example, ‘DFM’ outperforms ‘DFM-NCDEs’ on Korean GDP in terms of MSE, and ‘NCDEs’ performs better on UK GDP in terms of other metrics. I think contradicts to the assumption of the paper that DFM and NCDEs are complementary in achieving better performance?

Meta-review and decision
Meta-review by senior PC member, 2026-10-01
Promoted to phase 2

No

TheWebConf 2024 ✗ rejectedsubmitted 2023-10-31
110-22mean 0.4
reviews not public — shared by the authors download as .md

What the reviewers raised

  • Industry track; overall evaluations 1, 1, 0, -2, 2; rejected.
  • Reviewers asked for an ablation of the factor-extraction module, a VAE-based baseline, more countries, and a clearer account of how sudden shocks are handled.
Full reviews
Reviewer 1: rating 1 (weak accept)

Overall evaluation 1 (weak accept)

Overall evaluation (comments)

Summary:
The authors focus on solving the GDP nowcasting problem. They notice the drawbacks of the existing DFM model: the lack of capturing economic uncertainties and the limitation of capturing irregular dynamics. To deal with these issues, the authors combine neural controlled differential equations with DFM. They conduct experiments on two real GDP datasets. The results compared with enough competitors demonstrate the effectiveness of the proposed method.

Strong Points:
S1. The authors clearly demonstrate the solved problem and their motivations.
S2. The authors incorporate novel NCDE into classical DFM and propose a new nowcasting model, i.e., NCDENow.
S3. The authors conduct experiments on real datasets from SouthKorea and the UK. NCDENow shows better performance compared with 6 baselines in terms of nowcasting accuracy.

Weak Points:
W1. The authors argue that VAE models are like a black-box, which makes it challenging to maintain DFM's strengths (around in line 289) which seems to have no evidence to support this point from this paper. So, if the authors can cite some related works or add a VAE-based model as baseline, the statement will be more accurate.
W2. The combination of NCDE and DFM seems to be a naïve and straight solution to deal with the irregular time series forecasting problem.
W3. How is the issue mentioned in introduction, i.e., the lack of capturing economic uncertainties such as sudden recessions or booms, solved in NCDENow? I have no idea about this part. In my understanding about the time series, the emergent event such as Cov-19 cannot be predictable. Thus, it will cause unexpected changes of the observed time series, which is also unpredictable if there is no clear signal to indicate the influence of the event.
W4. The ablation study is not conducted. It seems that the factor extraction module can be solely used as predictor to forecast GDP. The influence of this module on the final prediction result is not verified. Moreover, there are lots of methods to obtain the latent factors, such as replacing factor extraction module with VAE. If the authors can conduct extensive experiments to demonstrate which solution is better, the whole manuscript will be more complete.

Reviewer 2: rating 1 (weak accept)

Overall evaluation 1 (weak accept)

Overall evaluation (comments)

The paper introduces a novel GDP nowcasting model that combines dynamic factor models and neural-controlled differential The paper introduces a novel GDP nowcasting model that combines dynamic factor models and neural-controlled differential equations. An experimental evaluation demonstrates its superior performance over six baselines, and the ability to provide effective insights into economic conditions for informed policy decisions. The paper reads well and the problem is relevant.

The main limitation is the application to only two datasets. Since GDP and economic data are usually provided as open data from governments and academics, I would have expected an experimental analysis considering more countries/macro-areas.

Reviewer 3: rating 0 (borderline paper)

Overall evaluation 0 (borderline paper)

Overall evaluation (comments)

This paper proposes a new framework called NCDENow for nowcasting Gross Domestic Product (GDP) growth. In this work, the authors investigate the traditional DFM model and try to fix its issues such as i) the lack of capturing economic uncertainties such as sudden recessions or booms; and ii) the limitation of capturing irregular dynamics from data of mixed frequency. To overcome the issues, authors proposed NCDENow that combines the strengths of Dynamic Factor Model (DFM) in extracting latent factors and Neural Controlled Differential Equations (NCDE) in handling irregular time series data. This paper claims that NCDENow outperforms existing methods in terms of nowcasting accuracy, especially in scenarios involving missing data and volatile economic conditions.

Strengths of this work:
1. This paper propose the idea that combines strengths of DFM and NCDE: NCDENow leverages the advantages of both DFM and NCDE, potentially leading to more accurate prediction of irregular time series data for GDP.
2. The proposed method improve handling of irregular data since NCDENow's use of NCDE allows it to better handle missing data and sudden fluctuations in economic indicators, often encountered in real-world data.
3. Form the experiments results, the paper achieves that NCDENow outperforms existing baselines in terms of nowcasting accuracy, particularly in challenging scenarios.
4. Interpretability is a merit. NCDENow provides insights into the relationships between extracted factors and GDP growth, aiding policymakers in making informed decisions.

Weaknesses of this work:
1. Some experimental details may be missing. The paper discuss the performance of proposed method but in the experiment setting hyperparameters and some setting details are not introduced. It will be better if authors can add these details.
2. Theoretical underpinnings: While the paper mentions well-posedness of NCDENow, a deeper discussion of the theoretical properties and justifications for the proposed framework would be beneficial.
3. The presentation of this work can be improved.

Reviewer 4: rating -2 (reject)

Overall evaluation -2 (reject)

Overall evaluation (comments)

This paper presents an innovative approach to predicting GDP and demonstrates commendable accuracy in forecasting the GDP of two specific countries. However, despite these strengths, the paper falls short in various critical aspects, leading to the overall recommendation of rejection.

The strengths of the paper: It introduces a new method for GDP prediction. The authors successfully implement this methodology to forecast the GDP of two countries, showcasing promising performance. The innovative approach and the accurate prediction of GDP for these countries highlight the potential of the proposed method in economic forecasting.

However, several weaknesses hinder the value to the WebConf audience. Firstly, the experiment section appears excessively restricted, failing to cater to the diverse interests of the audience. The paper should encompass broader applications such as predicting other time series in Economical / Financial domain. At the minimum, I suggest predicting GDP for multiple countries, particularly those in developing regions.

Secondly, the paper's method description section lacks relevance beyond the realm of GDP prediction. It fails to discuss the potential implications or adaptability of the proposed method to other time series prediction tasks. By extending the discussion to encompass its applicability in diverse forecasting domains, the paper could significantly increase its value to a broader to encompass its applicability in diverse forecasting domains, the paper could significantly increase its value to a broader audience.

Lastly, the absence of a rigorous analysis regarding the impact of individual modules within the proposed method is a notable weakness. The paper lacks ablation analysis or insights into the specific roles played by different modules. A deeper understanding of these components and their contributions to the forecasting process is crucial for comprehending the methodology's efficacy and potential enhancements.

In conclusion, while the paper introduces a novel method for GDP forecasting and demonstrates promising results for two specific countries, its limitations in scope, applicability, and lack of comprehensive analysis diminish its relevance and appeal to the broader audience of the WebConf.

Reviewer 5: rating 2 (accept)

Overall evaluation 2 (accept)

Overall evaluation (comments)

The paper introduces a novel NCDENow framework for GDP forecasting, merging neural-controlled differential equations with dynamic factor models, an innovative approach particularly effective during economic uncertainties. While the methodology and experimental evaluation are thorough, showing effectiveness against baseline models, the paper could benefit from more detailed technical explanations. Its significance in economic forecasting under uncertain conditions is highlighted, with potential impact on policymaking and economic analysis, though some technical aspects, particularly NCDE integration, warrant further clarity.

The paper does not delve deeply into the technical specifics of how Neural Controlled Differential Equations (NCDEs) are integrated with Dynamic Factor Models (DFMs). A more detailed explanation of the NCDEs' role and functioning within the framework would benefit readers, especially those unfamiliar with this approach.

While the paper presents results using datasets from South Korea and the United Kingdom, it does not extensively address the generalizability and scalability of the proposed method to other regions or larger datasets. Understanding how the model performs across diverse economic contexts and scales is crucial for its applicability in global economic forecasting.

The paper could enhance its analysis of how the proposed model handles sudden economic shocks and irregularities, such as those caused by unforeseen global events. While the COVID-19 pandemic is discussed, a broader examination of the model's robustness in various volatile scenarios would be beneficial.

KDD 2024 ✗ rejectedsubmitted 2024-02-02
2333mean 2.8
reviews not public on OpenReview — shared by the authors download as .md

What the reviewers raised

  • Applied Data Science track; overall evaluations 3, 3, 3 and 2; decision "revise and resubmit".
  • Reviewers asked for the design motivation of NCDENow, clearer training details, larger or more datasets, and a clearer distinction from plain NCDEs.
  • The meta-reviews questioned the applicability beyond GDP nowcasting and whether the method is used in practice.

What changed for the next submission

  • Reorganised the paper around six research questions and extended the discussion of limitations and future work.

KDD is an ACM SIGKDD conference; its OpenReview reviews are visible only to the authors and the committee. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”.

Full reviews and discussion
Reviewer kQMY: rating 3

Novelty 3, Presentation 3, Reproducibility 5, Overall evaluation 3

Paper category

3

Review

In the context of GDP nowcasting, the authors proposed a new framework named NCDENow. This framework first interpolate within the irregular datasets, followed by a NCDE neural networks to predict the coefficient of hidden states. Then, the time series data were sent to Kalman filter to extract the hidden variables. Combining the coefficients and hidden variables gives the desired predicted value. The author then compared the NCDENow with existing models on South Korea and UK GDP datasets. The results showed a significant decreases in MSE and MAPE in most of the studied cases. The NCDENow also showed robustness against data missing and high efficiency with further experiments.
The main part of this article is well-organized, the results demonstrated could answer the RQ proposed by the authors. The proposed outperformed other existing model. However, the design motivation/philosophy behind the NCDENow architecture is unclear. The model training should also be complemented. More experimental results is required to prove the advantages of the model.

Influence

3

Technical soundness

2

Impact

4

Summary

In the context of GDP nowcasting, the authors proposed a new framework named NCDENow. This framework first interpolate within the irregular datasets, followed by a NCDE neural networks to predict the coefficient of hidden states. Then, the time series data were sent to Kalman filter to extract the hidden variables. Combining the coefficients and hidden variables gives the desired predicted value. The author then compared the NCDENow with existing models on South Korea and UK GDP datasets. The results showed a significant decreases in MSE and MAPE in most of the studied cases. The NCDENow also showed robustness against data missing and high efficiency with further experiments.

Paper strength

1) The proposed framework of NCDENow is novel. It combines the advantages of DFM and NCDE in one framework to reach better results.
2) The result of NCDENow showed significant improvement of MSE and MAPE compared with existing models on South Korea and UK GDP dataset.
3) The NCDENow also showed higher efficiency and less parameters, while reached a better nowcasting performance.

Paper weakness

1) The design motivation/philosophy behind the NCDENow architecture is unclear. Is this architecture a pure empirically designed one?
2) The training process of NCDENow is not clear. For instance, how to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models?
3) The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired.

Detailed evaluation and suggestions for authors

1) A Question on the NCDENow framework. As described by the author, the "regression step" does not train directly the coefficients in this step, so it may cause some confusion if this step is termed "regression".
2) The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired.

Questions for authors for discussion

1) How to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models?
2) A question considering the results in table 8: the MAPE seems to be large (57%) even for NCDENow for the sudden drop. How to comment on this result?
3) What is the experiment setting for RQ3?

Confidence level

1

Authors, 2024-04-12
Rebuttal

We sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions:

---

Q1. How to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models?

(1) Updating neural network parameters

For training the neural network, please refer to Algorithm 1 in Section 3.5, where line 11 describes updating the parameters.

(2) Labels to train models

If the GDP growth rate we want to nowcast is at time \(N+1\), as shown in Figure 3, we use the GDP growth rate up to time \(N\) and related economic indicators up to time \(N+1\) for training. The label (ground truth) you mentioned is the GDP growth rate at time \(N+1\). This learning method is about the concept of nowcasting; please refer to Figure 2.

Q2. A question considering the results in table 8: the MAPE seems to be large (57%) even for NCDENow for the sudden drop. How to comment on this result?

We apologize for the confusion. There was an error in the calculation of the MSE and MAPE for the COVID-19 period (2020Q1, 2020Q2, 2020Q3) that caused the 2020Q3 forecast to be missing. As shown in the table below, the corrected MAPE for NCDENow decreases to 53.87%, while the MAPE for DFM increases to 99.55%. The hyperparameters are the best settings based on the MSE, which are the same as those described in Table 3.

| COVID-19 | Ground Truth | NCDENow | DFM |
|------|:-----:|:------:|:-------:|
| 2020Q1 | -1.2615067 | 0.1034569 | 0.10668433 |
| 2020Q2 | -3.1507382 | -3.3880901 | -1.14140557 |
| 2020Q3 | 2.2334244 | 1.2085018 | -0.5897278 |
| MSE | - | 0.9900 | 4.6265 |
| MAPE | - | 0.5387 | 0.9954 |

The best setting based on MAPE (learning rate: 0.01, hidden dimension: 512, # layers: 3) results in a performance of 44.53%, which you could also consider poor. However, we can see from this study that the performance can be significantly improved compared to DFM, which is used by several central banks for GDP nowcasting.

Q3. What is the experiment setting for RQ3?

The experimental setting for RQ3 is identical to that shown in Table 5.

Authors, 2024-04-12
Rebuttal

W1. The design motivation/philosophy behind the NCDENow architecture is unclear. Is this architecture a pure empirically designed one?

Our architecture is not an empirical construct but a fusion of DFM and NCDE, drawing from a theoretical background[1,2].

The novelty of NCDENow lies in integrating DFM with NCDE's advanced handling of complex, non-linear dynamics. This integration is motivated by the recognition that traditional DFMs, while powerful for linear analysis, struggle to capture complex patterns and rapid economic changes. NCDEs, generalizing over RNNs, offer a more expressive framework.

Our strategic design aims to synergize the interpretability of DFMs with the temporal modeling capabilities of NCDEs. This is not an ad-hoc combination but a thoughtful response to the limitations of existing nowcasting models, with a clear goal to enhance predictive accuracy and adaptability. Also, our NCDENow can incorporate diverse DFM methods, highlighting the extensibility of NCDENow.

> [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024).
>
> [2] Kidger, Patrick, et al. “Neural controlled differential equations for irregular time series.” NeurIPS. 2020.

W2. The training process of NCDENow is not clear. For instance, how to update the parameters of the neural networks? Would yt (t<=N) be used as labels to train models?

Please refer to our response for Q1.

W3. The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired.

Since the release frequency of the economic indicators used in our project is monthly or quarterly, the size of the dataset is not sufficient. Therefore, in future work, we will consult with the Korean government agencies involved in this project to find a way to use high-frequency data.

Authors, 2024-04-12
Rebuttal

S1. A Question on the NCDENow framework. As described by the author, the "regression step" does not train directly the coefficients in this step, so it may cause some confusion if this step is termed "regression".

We apologize for the confusion. We will change the terminology to "fusion module" in our updated version.

S2. The sizes of the dataset seemed to be insufficient. Only two small datasets were tested. I suppose that this is why only one layer of neural networks is enough to reach the best MSE and MAPE. More cases are desired.

Please refer to our response for W3.

Authors, 2024-04-19
Comment

Dear Reviewer kQMY, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer.

Reviewer tRk4: rating 3

Novelty 4, Presentation 3, Reproducibility 5, Overall evaluation 3

Paper category

4

Review

The paper is technically dense but well-structured, presenting a clear methodology and substantial experimental validation. The originality of combining DFMs with NCDEs for nowcasting GDP represents a significant contribution to the literature, addressing limitations in capturing economic uncertainties and irregular dynamics. Nonetheless, the paper could have better contextualized its contributions within the existing body of work.

While the proposed NCDENow framework shows potential in improving nowcasting accuracy and handling irregular time-series data, its significance is somewhat overshadowed by the lack of demonstrated real-world deployment or impact.

Pros
- Innovative integration of methodologies.
- Demonstrated real-world applicability and superior performance.
- Extensive experimental evaluation and comparison.

Cons
- Limited theoretical depth and discussion on assumptions.
- Focus on specific datasets limits generalizability.
- Complexity of the approach may hinder broader accessibility.

Influence

3

Technical soundness

5

Impact

5

Summary

The paper presents NCDENow, a promising framework that leverages the strengths of DFMs and NCDEs for GDP nowcasting, aiming to address challenges related to economic uncertainties and data irregularities. While it demonstrates methodological innovation and improved forecasting accuracy through experimental validation, it falls short in evidencing real-world deployment and impact, limiting its significance within the applied data science context.

Paper strength

- The paper presents a novel approach by combining NCDEs with traditional DFM, enhancing the model's capability to handle irregular time series data and capture economic uncertainties.
- Extensive experiments conducted on real-world datasets from South Korea and the United Kingdom, demonstrating the model's superior performance over existing benchmarks.
- By addressing the specific challenge of GDP nowcasting, the paper contributes valuable insights that can assist policymakers and analysts in making informed decisions based on timely and accurate economic indicators.

Paper weakness

- The paper could provide a deeper theoretical foundation for the choice of integrating NCDEs with DFM, explaining why this combination is particularly suited for addressing the challenges of nowcasting GDP.
- While the model shows promising results on the datasets used, the paper could benefit from a discussion on its applicability to other types of economic data or different geographic regions.
- More detailed information on model architecture, hyperparameters, and training details could enhance the reproducibility of the results. Additionally, discussing how the model performs with varying amounts of available data could provide insights into its robustness.

Detailed evaluation and suggestions for authors

- Authors are encouraged to provide more theoretical insights into the integration of NCDEs with DFM, expand on the model's applicability to other economic indicators or regions.
- Conducting additional experiments across more diverse economic contexts.
- A comparative analysis with existing models across a broader range of economic conditions would strengthen the paper's contributions.
- Enhance the paper's reproducibility through more detailed model documentation and analysis on data variability.

Questions for authors for discussion

- Can the authors provide additional theoretical background on the integration of NCDEs with DFM?
- Could you elaborate on the robustness of the model in scenarios with varying data availability or quality?
- Could the authors detail the preprocessing steps for handling irregular time series data, particularly any data imputation or transformation techniques employed before applying the NCDE-DFM framework?
- Has any sensitivity analysis been conducted to assess the impact of key parameters on the forecasting performance of NCDENow? How sensitive is the model to changes in its architecture or hyperparameters?
- Are there limitations or challenges in applying NCDENow that were not discussed, and how might these affect its broader applicability?

Confidence level

4

Authors, 2024-04-12
Rebuttal

We sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions:

---

Q1. Can the authors provide additional theoretical background on the integration of NCDEs with DFM?

Below, we describe the theoretical background of the integration of NCDE and DFM.

The DFM is expressed in state-space form and can be estimated using Kalman filters and multiple solution algorithms. While DFMs have been well studied on data using economic indicators, they may be insufficient to model complex patterns or rapid changes in data due to the lack of non-linear activation. A possible complement to this is the use of Neural CDEs, which can theoretically generalize not only RNNs but also state-space models [1], which have recently gained attention.
Thus, our NCDENow is a framework that utilizes both the advantages of NCDEs, which are more expressive than traditional RNN-based models [1], and the advantages of DFMs, which are complementary to each other.

Also, our NCDENow has the advantage of being an extensible framework that can be extended to other kinds of DFM methods in addition to the DFM we used.

> [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024).

Authors, 2024-04-12
Rebuttal

Q2. Could you elaborate on the robustness of the model in scenarios with varying data availability or quality?

(1) Robustness of the model with respect to data quality

To check the robustness of NCDENow with respect to data quality, we performed GDP nowcasting after arbitrarily removing a portion of the economic indicators (10% and 20%). The results are shown in Table 7. Even with poor data quality, NCDENow outperformed DFM, the leading GDP nowcasting model used by many central banks around the world.

(2) Robustness of the model with respect to data availability

We used economic indicators from South Korea and the UK to nowcast the GDP growth rate of each country. The South Korean economic indicators were available through a web service provided by the Bank of Korea [1]. However, some of the UK economic indicators used in previous work [2], such as the PMI construction and the UK focused equity index, were unavailable due to their proprietary nature. Nevertheless, in both scenarios, NCDENow outperforms all baseline models (including DFM) in terms of MSE and MAPE, demonstrating the robustness of NCDENow with respect to data availability.

In the future, we will consult with government agencies in South Korea that are participating in this project to find ways to use high-frequency data collected by financial authorities, which will allow us to conduct various experiments on quantitative data. In addition, our project is considering applying NCDENow to exchange rate forecasting to enhance its usefulness. We will provide more details on this in our response to question 5.

> [1] Bank of Korea: Economic Statistics System
>
> [2] Anesti, Nikoleta, Ana Beatriz Galvão, and Silvia Miranda-Agrippino. "Uncertain Kingdom: nowcasting GDP and its revisions." (2018).

Authors, 2024-04-12
Rebuttal

Q3. Could the authors detail the preprocessing steps for handling irregular time series data, particularly any data imputation or transformation techniques employed before applying the NCDE-DFM framework?

(1) Data imputation

The exposure estimation module uses the cubic spline method. This interpolation method imputes missing values or outliers by connecting data points with a curve between them. It has the advantage of preserving the natural dynamics of time series.

(2) Data transformation

The factor extraction module uses two different types of transformation methods (differencing or log differencing), depending on the nature of the economic indicator. For some indicators, such as GDP growth, we use the raw values. In our paper, Table 2 shows the transformation method for each economic indicator.

- Differencing: \(\Delta D_n = D_n - D_{n-1}\)
- Log Differencing: \(\Delta \log(D_n) = \log(D_n) - \log(D_{n-1})\)

Authors, 2024-04-12
Rebuttal

Q4. Has any sensitivity analysis been conducted to assess the impact of key parameters on the forecasting performance of NCDENow? How sensitive is the model to changes in its architecture or hyperparameters?

(1) ODE Solver

In our paper, Table 9 shows the GDP nowcasting results when using the Euler method and the 4th order Runge-Kutta (RK4) method. For all datasets, the RK4 method performs better than the Euler method because it performs more precise operations. For example, the Euler method performs only one computation at each time step, while the RK4 method performs four computations and then uses the average value. The RK4 method is known to be more accurate for solving nonlinear differential equations [1]. Because it is specifically designed for various types of differential equations, it provides better accuracy than the simplest method, the Euler method.

> [1] Islam, Md Amirul. "A comparative study on numerical solutions of initial value problems (IVP) for ordinary differential equations (ODE) with Euler and Runge Kutta Methods." American Journal of computational mathematics 5.03 (2015): 393-404.

(2) Number of hidden layers

In our paper, Table 10 shows how the performance of NCDENow varies depending on the number of hidden layers in the exposure estimation module. For the South Korea dataset, the lowest MSE was achieved when using a single layer, and for the UK dataset, the lowest MSE was obtained when using three layers.

(3) Dimension of hidden layer

Although we could not include it due to the page limit, we also carried out a sensitivity analysis on the dimension of the hidden layer. The results are as follows:

| | Korean GDP | | UK GDP | |
|:---:|:----------:|:------:|:------:|:------:|
| D | MSE | MAPE | MSE | MAPE |
| 4 | 0.6390 | 0.4763 | 0.0186 | 0.2312 |
| 8 | 0.6398 | 0.5803 | 0.0121 | 0.2304 |
| 16 | 0.5987 | 0.6582 | 0.0168 | 0.2024 |
| 32 | 0.5620 | 0.5096 | 0.0145 | 0.2359 |
| 64 | 0.6164 | 0.6193 | 0.0146 | 0.2171 |
| 128 | 0.5104 | 0.7049 | 0.0142 | 0.1988 |
| 256 | 0.5191 | 0.5808 | 0.0070 | 0.1385 |
| 512 | 0.5552 | 0.8736 | 0.0146 | 0.1979 |

The South Korea dataset had the lowest MSE when the size of the hidden dimension was 128, and the UK dataset had the lowest MSE and MAPE when the size of the hidden dimension was 256.

Authors, 2024-04-12
Rebuttal

Q5. Are there limitations or challenges in applying NCDENow that were not discussed, and how might these affect its broader applicability?

(1) Nowcasting of other economic indicators

We are currently considering expanding the use of the NCDENow model to nowcast the KRW-USD exchange rate. Economic experts are involved in collecting various daily or monthly economic indicators that affect the exchange rate. For example, the economic indicators relevant to the KRW-USD exchange rate are shown in the following table.

| Group | Economic indicators |
|:-------------------:|:--------------------------------------------------------------------:|
| U.S. Interest Rates | 1-year / 2-year / 10-year treasury bond yield |
| South Korea Interest Rates | 1-year / 2-year / 10-year treasury bond yield, 3-year corporate bond |
| South Korea Stock Market | Cumulative net purchase amount by foreigners in KOSPI |
| Commodities | WTI (West Texas Intermediate crude oil price) |
| Exchange Rate | Dollar index, KRW-USD exchange rate |

We are also working on structuring appropriate factor groups considering the characteristics of each economic indicator. Once this work is completed, we expect NCDENow to be used to nowcast the KRW-USD exchange rate.

(2) GDP Nowcasting for other countries

As mentioned in Section 5, a limitation of our study is that we focused only on South Korea and the United Kingdom. Therefore, in the future, we will extend our experiments to nowcast GDP growth for a variety of countries. In particular, we will consider datasets from countries whose economic indicators tend to be volatile.

(3) Model Architecture

In NCDENow, the factor extraction module based on the Kalman filter and smoother extracts factors by economic indicator group, and the exposure estimation module based on NCDE estimates factor exposures and idiosyncratic disturbances. As mentioned in Section 5, we will attempt to use the mechanisms of NCDE to model the continuous modelling of the factor estimation process within the Kalman filter and smoother. Through this work, we hope to explore the potential of the NCDENow model for GDP nowcasting.

Authors, 2024-04-12
Rebuttal

W1. The paper could provide a deeper theoretical foundation for the choice of integrating NCDEs with DFM, explaining why this combination is particularly suited for addressing the challenges of nowcasting GDP.

Please refer to our respones for Q1.

W2. While the model shows promising results on the datasets used, the paper could benefit from a discussion on its applicability to other types of economic data or different geographic regions.

Please refer to our respones for Q5.

W3. More detailed information on model architecture, hyperparameters, and training details could enhance the reproducibility of the results. Additionally, discussing how the model performs with varying amounts of available data could provide insights into its robustness.

(1) Model architecture

Please refer to our respones for Q1.

(2) Hyperparameter settings

Please refer to our respones for Q4.

(3) Amounts of data

As the data collected in this project are monthly/quarterly economic indicators, the amount of time series data is not abundant. Therefore, we will consult with Korean government agencies participating in this project to find ways to use high-frequency data, and we will conduct various experiments regarding the amount of data in the future.

Authors, 2024-04-19
Comment

Dear Reviewer tRk4, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer.

Reviewer 5y5M: rating 3

Novelty 3, Presentation 3, Reproducibility 4, Overall evaluation 3

Paper category

1

Review

The research is interesting and meaningful. The quality is good and the manuscript has some innovations.
The proposed model is based on existing related researches, and the innovations should be further illsturated to demonistrate the contribution of the research in a clearer way.
There are many other researches that can detect abnormal fluctuations in large-scale time series, the research should discuss the unique advantages of the proposed model between thoses researches.
The illustration of the research could be further improved. For example, the mechanism of NCDE to detect idiosyncratic disturbances; The domain expert knowledge data for factor group information data and etc.

Influence

2

Technical soundness

4

Impact

3

Summary

DFM suffers from two issues: i) the lack of capturing economic uncertainties such as sudden recessions or booms; and ii) the limitation of capturing irregular dynamics from data of mixed frequency. To overcome the above two challenges, the research introduce a new GDP Nowcasting framework called NCDENow , which combines neural controlled differential equations (NCDEs) to DFM, which effectively handles the dynamics of irregular time series. NCDENow consists of two main parts: i) one extracts factors reflecting the economic conditions based on DFM, and ii) the other captures temporal dependencies of the impact of these factors on GDP growth based on NCDE. The research evaluates NCDENow against 6 baselines on 2 real-world GDP datasets from South Korea and United Kingdom, demonstrating its enhanced predictive capability. The empirical results favor our method, highlighting the significant potential of integrating NCDE into nowcasting models.

Paper strength

The research is meaningful and interesting. The method of combining DFM with NCDEs is novel. The experiment is solid and can better support the effectiveness of the proposed model.

Paper weakness

The proposed model is based on existing related researches, and the innovations should be further illsturated to demonistrate the contribution of the research in a clearer way.
There are many other researches that can detect abnormal fluctuations in large-scale time series, the research should discuss the unique advantages of the proposed model between thoses researches.
The illustration of the research could be further improved. For example, the mechanism of NCDE to detect idiosyncratic disturbances; The domain expert knowledge data for factor group information data and etc.
The research is suggested to add new content to evaluate the effectivenss of the proposed model in real applications.

Detailed evaluation and suggestions for authors

Please see the content in paper weakness.

Questions for authors for discussion

Please see the content in paper weakness.

Confidence level

3

Authors, 2024-04-12
Rebuttal

We sincerely thank you for the review and positive feedback. Below, we address each of your concerns and questions:

W1. The proposed model is based on existing related researches, and the innovations should be further illsturated to demonistrate the contribution of the research in a clearer way. There are many other researches that can detect abnormal fluctuations in large-scale time series, the research should discuss the unique advantages of the proposed model between thoses researches. The illustration of the research could be further improved. For example, the mechanism of NCDE to detect idiosyncratic disturbances; The domain expert knowledge data for factor group information data and etc. The research is suggested to add new content to evaluate the effectivenss of the proposed model in real applications.

(1) The contribution of the research

GDP nowcasting has been studied based on DFM models. Our novelty is that to the best of our knowledge, no research has been done to integrate Neural CDEs, which can generalize recurrent neural networks, with DFMs, which have strengths in economic indicator data. Our contribution is that our NCDENow framework can be used in conjunction with many other variants of DFMs. This implies a promising extensibility of NCDENow, which is our strength.

We would like to explain that time series abnormal detection models and NCDENow have different objectives as you mentioned. The main difference between NCDENow and time series abnormal detection models is their respective goals and applications. While time series abnormal detection models are designed to identify unusual patterns or outliers in the data that may indicate errors or new events, NCDENow was developed specifically for GDP nowcasting. Our model aims to accurately nowcast current economic conditions using a combination of DFM and NCDE, and focuses on capturing and interpreting the underlying economic dynamics rather than detecting anomalies. In addition, unlike anomaly detection models that focus on pattern recognition and identifying deviations, NCDENow leverages NCDE to model the ongoing dynamics of economic indicators, providing a deeper understanding of the temporal dependence and the impact of macroeconomic factors on GDP growth.

And when we say that we are good at capturing abrupt fluctuations in Figure 1, we mean that DFM is a linear state-space model, whereas NCDE has the advantage of compensating for the non-linearity of these fluctuations.

Authors, 2024-04-12
Rebuttal

*The following continues from the previous rebuttal for W1.*

(2) The mechanism of NCDE to detect idiosyncratic disturbances

As shown in Eq. (4), the exposure estimation module in NCDENow uses an augmented NCDE to estimate factor exposures and idiosyncratic disturbances. The module uses the cubic spline method to generate continuous paths from economic indicators. From these paths, the NCDE continuously learns how economic indicators change over time and identifies and captures idiosyncratic disturbances, which are unexpected fluctuations. This mechanism effectively detects idiosyncratic disturbances in complex time series, enabling more precise GDP nowcasting.

(3) The economic domain expert knowledge

The dataset of economic indicators for South Korea was selected, taking into account its economic structure. In South Korea, manufacturing and imports and exports account for a large share of the economy, and we included economic indicators such as manufacturing production and shipments, and the size of exports and imports, as in previous work [1]. We have also classified the economic indicators into four blocks (global, real, soft, and labor) as in the previous work [1]: the global group includes all economic indicators; the real group includes indicators related to the real economy; the soft group includes survey data, considering the importance of soft indicators in GDP nowcasting [2]; the labor group consists of indicators related to the labor market.

In our project, we are constantly analyzing the characteristics of the economic indicators in order to update the dataset. For example, we have recently been working with economic experts to correct measurement errors in the ‘Number of job openings per unemployed person’ indicator.

We hope our response will address your concerns.

> [1] Hyun Chang Yi, Dongkyu Choi, and Yonggun Kim. 2022. Dynamic Factor Model and Deep Learning Algorithm for GDP Nowcasting. Issue Note 28 (2022).
>
> [2] KURZ‐KIM, Jeong‐Ryeol. A note on the predictive power of survey data in nowcasting euro area GDP. Journal of Forecasting, 2019, 38.6: 489-503.

Authors, 2024-04-19
Comment

Dear Reviewer 5y5M, as the discussion period comes to a close, we would like to thank you once again for your positive assessment. Your support and comments have been invaluable to us. We remain open and eager to incorporate any further feedback or insights you might offer.

Reviewer bngv: rating 2

Novelty 2, Presentation 3, Reproducibility 4, Overall evaluation 2

Paper category

4

Review

The paper proposes a novel framework called NCDENow, which integrates dynamic factor models (DFM) with neural controlled differential equations (NCDE) for GDP nowcasting. It is well-written and easy to understand, and the problem of nowcasting GDP is important. However, my major concern is that the novelty of the proposed method is insufficient.

Influence

2

Technical soundness

4

Impact

3

Summary

The paper proposes a novel framework called NCDENow, which integrates dynamic factor models (DFM) with neural controlled differential equations (NCDE) for GDP nowcasting. While the methodology demonstrates improved performance in handling irregular data and capturing economic uncertainties compared to baselines, it lacks novelty in its extension of DFM and relies heavily on NCDE for addressing irregular dynamics. Further, the paper could benefit from presenting more innovative designs built upon NCDE.

Paper strength

1. This paper is well-written and easy to understand.
2. The problem of nowcasting GDP is important.
3. Experiments are condudcted by comparing with six baselines.

Paper weakness

1. The noverty of the proposed method is limited.
2. The proposed method does not significantly distinguish itself from NCDE.
3. Discussion on design Innovations is not sufficient.

Detailed evaluation and suggestions for authors

1. The proposed method seemly simply integrate NCDE with DFM. The extension seems relatively straightforward, lacking significant innovation beyond leveraging NCDE's capabilities.

2. The paper heavily relies on NCDE for handling irregular dynamics, without introducing substantial enhancements or innovations within the NCDE framework itself.

3. The paper lacks in-depth discussions on innovative designs or adaptations within the NCDE framework that could further distinguish NCDENow from existing approaches, potentially limiting its novelty and significance.

Questions for authors for discussion

1. Can you elaborate on any novel designs or adaptations within the NCDE framework that differentiate NCDENow from existing approaches? How does NCDENow leverage the unique capabilities of NCDE beyond standard applications?
2. Given the comment about the reliance on NCDE for addressing irregular dynamics, are there additional enhancements or innovations within the NCDE framework that could further strengthen the proposed method's performance and novelty?
3. How does NCDENow address potential limitations or challenges associated with the integration of DFM and NCDE? Are there specific considerations or strategies employed to ensure the synergy and effectiveness of these two components in the nowcasting process?

Confidence level

2

Authors, 2024-04-12
Rebuttal

We sincerely thank you for the review and feedback. Below, we address each of your concerns and questions:

---

Q1. Can you elaborate on any novel designs or adaptations within the NCDE framework that differentiate NCDENow from existing approaches? How does NCDENow leverage the unique capabilities of NCDE beyond standard applications?

Our novelty is the integration of the DFM with the NCDE framework. NCDENow can be seen as a unification of CDEs that generalize recurrent models. It can also be linked to linear CDEs, which can theoretically generalize state-space models[1] (e.g., S4[2], Mamba[3]), which have recently gained prominence. Thus, NCDENow encompasses traditional recurrent models and, according to [1], is more expressive than traditional recurrent models, while also leveraging the strengths of DFMs.

As you know, our NCDENow is based on the NCDE framework, and we had the design goal of integrating it with DFM as a loose dependency. We plan to integrate the Kalman filter of a DFM into NCDE in our future work.

The advantage of our NCDENow is that it is a framework that can use not only the DFM we used, but also other kinds of DFM methods, and our method is extensible in this respect.

> [1] Cirone, Nicola Muca, et al. "Theoretical Foundations of Deep Selective State-Space Models." arXiv preprint arXiv:2402.19047 (2024).
>
> [2] Gu, Albert, Karan Goel, and Christopher Ré. "Efficiently modeling long sequences with structured state spaces." arXiv preprint arXiv:2111.00396 (2021).
>
> [3] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).

Q2. Given the comment about the reliance on NCDE for addressing irregular dynamics, are there additional enhancements or innovations within the NCDE framework that could further strengthen the proposed method's performance and novelty?

Our NCDENow focuses on integrating the respective strengths of NCDE and DFM. Although we mention it as future work in the conclusion section, our goal of NCDENow in this paper is not our intention to replace the functions of NCDE with the algorithms of DFM. Instead, our NCDENow has the advantage of being able to use some kind of DFM with NCDE, which can generalize recurrent models.

Authors, 2024-04-12
Rebuttal

Q3. How does NCDENow address potential limitations or challenges associated with the integration of DFM and NCDE? Are there specific considerations or strategies employed to ensure the synergy and effectiveness of these two components in the nowcasting process?

NCDE has the advantage of effectively modeling non-linear and irregular time series data, but it may not sufficiently reflect economic theory. On the other hand, DFM has the advantage of capturing correlations between economic indicators well, but is limited by linearity and normality assumptions. Utilizing the strengths and weaknesses of each of these models to complement each other is an effective approach to improving overall prediction performance.

As a strategy for this, the NCDENow framework clearly distinguishes between the roles of DFM and NCDE. DFM extracts latent factors by taking advantage of the ability to capture correlations between economic indicators, and NCDE models non-linear and irregular characteristics well to derive idiosyncratic disturbances and factor loadings. By integrating the results derived from these two modules in the regression module, the advantages of DFM and NCDE can be effectively combined to improve overall prediction performance.

This design is expected to maximize the synergy effect between NCDE and DFM and enable effective GDP nowcasts that reflect both economic theory and non-linear dynamics.

W1. The novelty of the proposed method is limited.

Please refer to our response for Q1.

W2. The proposed method does not significantly distinguish itself from NCDE.

Please refer to our response for Q2.

W3. Discussion on design Innovations is not sufficient.

Please refer to our response for Q3.

Authors, 2024-04-19
Comment

Dear Reviewer bngv, as the discussion period comes to a close, we would like to thank you once again for your insightful comments have been invaluable to us. We remain open and eager to incorporate any further questions or suggestions you might offer.

Meta-review and decision
Meta Review by Program Chairs, 2024-05-14

Recommendation 1, Confidence 3

Meta review

The reviewers largely agree on the overall merit of the work but also point at weaknesses. Our primary concern is the very limited applicability of the insights from this research outside the very specific application to GDP prediction. For the specific goal of GDP nowcasting for the purpose of policy adjustments, it is unclear whether the approach is in fact used as of today and the value those improvements may have had. For an applied paper, some more color around the utility and impact of a MAPE/MSE reduction by 50% would be important - as well as a justification of either MAPE or MSE as the most relevant metric given that both assume symmetric impact of both over and underprediction. All those application relevant nuances are unfortunately not presented.

Paper category

1

Meta Review by Area Chair JPEo, 2024-05-14

Recommendation 1, Confidence 2

Meta review

Reviews are quite heterogeneous (both in terms of reviewer confidence and specific questions/observations). Overall, there is good consensus on the relevance on the problem, and appreciation for the solid experimentation methodology and evaluation against multiple baselines (although in this specific case the available real world datasets are theoretically too small for an accurate evaluation of the proposed algorithm). The paper is well written and all the questions of the reviewers have been answered.
As noted by one reviewer, the paper may be slightly out-of-scope for the ADS track, since it shows good quality results validated on real GDP data of two specific countries, but it falls short in evidencing significant real-world deployment and impact, and does not introduce a benchmark dataset.

Paper category

3

Decision by Program Chairs, 2024-05-16
Decision

Revise and Resubmit

CIKM 2024 ✓ acceptedsubmitted 2024-05-13, decision 2024-07-16
1010mean 0.5
reviews not public — shared by the authors download as .md

What was discussed

  • Applied Research track; overall recommendations 1, 0, 1, 0; accepted.
  • Remaining requests: stronger and more recent nowcasting baselines, dataset statistics, and validation on more countries.

CIKM is an ACM conference and does not publish reviews. ACM’s Peer Review Policy FAQ keeps reviewer identities and review content confidential, but states that “authors are permitted to use and share the text of the reviews that they receive”.

Full reviews
Reviewer 1: rating 1 (weak accept)

Overall recommendation 1 (weak accept)

Quality of Presentation

4 (good)

Detailed Comments to the Author(s)

Strengths:
[+] The presentation is easy to follow. Most figures are well-designed and deliver concrete information.
[+] The limitations and future studies are discussed at the end of this manuscript.
[+] The experimental section is solid, with 6 research questions and concrete evaluations.

Major weaknesses:
[-] In Table 4, the proposed model outperforms SoAs significantly, which is great. However, the baselines seem too easy and old. There are many advanced models for forecasting and nowcasting, with time series data or unstructured data. The authors need to justify why they are not compared.
[-] In Line 512, authors mention there is a theoretical analysis of the proposed method, which can be linked to other methods. However, I didn’t find corresponding sections in the manuscript. Maybe authors mean the content around Line 362?
[-] In Table 6, the proposed model did well during COVID-19. Explanations are needed.
[-] The dataset statistics needs to be introduced for better understanding. For example, the partition of training, testing datasets. Differences and similarities between two datasets.
[-] The writing can be further polished. For example, “want to” is typically not often used in scientific papers.
[-] Also, there are some typos. For example, a period is missing in the caption of Table 5, Table 7, and Figure 5.

Summary to support your recommendation

This study nowcasts GDP under normal and abnormal periods. The problem is important and has great social impact. The authors design NCDENow, which integrates the advantage of DFM and NCDE. NCDENow outperforms baseline models significantly on datasets from South Korea and the UK. The main concerns are regarding the experimental design and baseline comparison. Also some important justification and explanation. Another concern is the missing of deployment or system implementation section, which are encourages by the Applied Research Track.

Reviewer 2: rating 0 (borderline paper)

Overall recommendation 0 (borderline paper)

Quality of Presentation

5 (excellent)

Detailed Comments to the Author(s)

The author shows the model's performance in the UK and South Korea and releases the codes. Why not show more countries that are impacted by irregular events like COVID-19 as well? This would help to check if the model runs robustly.

Summary to support your recommendation

The paper introduces a novel nowcasting framework, NCDENow, that combines Dynamic Factor Models (DFM) with Neural Controlled Differential Equations (NCDE) to enhance GDP prediction. This hybrid approach aims to address the shortcomings of traditional DFMs by effectively capturing the dynamics of irregular time series data, providing significant improvements in nowcasting accuracy and interpretability compared to existing methods. The method's effectiveness is validated through empirical results on real-world GDP datasets from South Korea and the United Kingdom.

Reviewer 3: rating 1 (weak accept)

Overall recommendation 1 (weak accept)

Quality of Presentation

5 (excellent)

Detailed Comments to the Author(s)

Strong points:
- The paper clearly identifies the challenges in GDP nowcasting, particularly the limitations of Dynamic Factor Models (DFMs). This sets a solid foundation for understanding the proposed solution.
- The paper presents a novel approach to address the identified issues as the first research to integrate Neural Controlled Differential Equations (NCDEs) with DFMs for economic indicators. The architecture to integrate DFMs with NCDEs is simple to understand and computationally efficient for an initial attempt.
- The paper is generally well-written and includes step-by-step numerical experiments to justify the authors' claims.

Weak points:
- While the performance is superior in scenarios with irregular data and volatile economic conditions, the datasets used for validation seem limited, as the authors also stated. For instance, the ability in the volatile conditions was demonstrated using only one instance: COVID-19 in the Korean economy. Including datasets from a more diverse set of countries could provide a better assessment of the model's generalizability across different economic conditions.
- The selection of baselines is limited to variations of RNN-based models, and all RNN-based baselines rely on DFM for data preprocessing. Including a broader range of baseline models would provide a more comprehensive comparison.

Question:
Could the proposed methodology be applied to other macroeconomic metrics in similar situations, such as inflation forecasting, unemployment rate prediction, or industrial production nowcasting? Has this been verified?

Summary to support your recommendation

The paper offers a valuable contribution to the field of GDP nowcasting by identifying current challenges and proposing a novel framework that integrates Neural Controlled Differential Equations with Dynamic Factor Models. The paper is well-written, proposes initiating solution, and includes detailed numerical experiments. However, due to the limitations regarding dataset validation and the selection of baselines, I recommend a weak acceptance.

Reviewer 4: rating 0 (borderline paper)

Overall recommendation 0 (borderline paper)

Quality of Presentation

3 (fair)

Detailed Comments to the Author(s)

This paper proposes a novel framework for GDP nowcasting using irregularly sampled macroeconomic indicators, based on the Neural Controlled Differential Equations framework for modeling latent factors. The authors compare their approach against DFM and its combination with different flavors of sequential models. Several research questions are explored by empirical evaluation on two datasets, including prediction performance, effects of missing data and ODE solvers used.

Strong points:
* Quite fresh and interesting idea to explore new generalization of both sequential and state-space models in the problem of nowcasting (or filtering) the GDP
* Based on the error metrics the proposed method outperforms the alternatives (although a weak ones) by a high margin (though there are some unexpected patterns - more about it in the weak points section)
* Easy to follow manuscript and detailed description of algorithm/modeling. Also contributed code and the dataset

Weak points:
* Weak comparison against the alternative approaches. Mostly using general sequence approaches in combination with DFM, instead of comparing against contemporary specialized GDP-nowcasting approaches mentioned in references like [26] and [40], or others non cited works like:
- Richardson, Adam, Thomas van Florenstein Mulder, and Tuğrul Vehbi. "Nowcasting GDP using machine-learning algorithms: A real-time assessment." International Journal of Forecasting 37, no. 2 (2021): 941-948.
- Eraslan, Sercan, and Maximilian Schröder. "Nowcasting GDP with a pool of factor models and a fast estimation algorithm." International Journal of Forecasting 39, no. 3 (2023): 1460-1476.
* In the Table 4 for South Korea GDP nowcasting, it is a bit unexpected to see that among the baselines DFM has the smallest MSE, but has the highest MAPE. Some comments/discussion on that pattern would be appreciated, especially since that pattern is not present in the results for UK GDP.
* The RQ2 is narrowed to just comparing DFM and NCDENow, while it would be interesting to see the results for all the alternative approaches. Also the conjecture/comment was that NCDENow seems “more robust to missing data” but from Table 5 it seems that 20% missing data is actually improving DFM performance for both South Korea and UK tasks (reduced MSE and MAPE compared to no missing data in Table 4.). And there was also no comments/discussion on that quite surprising pattern.

Summary to support your recommendation

Even though the idea seems novel and the results look interesting, it seems that the model is not deployed in the real world.

Meta-review and decision
Meta-review by senior PC member, 2024-07-16
Meta-review

The paper presents NCDENow, a novel nowcasting framework that integrates Dynamic Factor Models (DFM) with Neural Controlled Differential Equations (NCDE) to enhance GDP prediction. Despite some areas for improvement, the paper's strengths justify its acceptance.

1. Practical Relevance and Novelty:
The paper addresses the critical and timely issue of GDP nowcasting, particularly under irregular and volatile economic conditions. The integration of NCDE with DFM is an innovative approach that significantly improves nowcasting accuracy and interpretability compared to existing methods. This novelty is well-recognized by multiple reviewers, emphasizing its potential impact on economic forecasting.

2. Comprehensive Experimental Design:
The authors conducted thorough experiments using real-world GDP datasets from South Korea and the United Kingdom. The experimental design is robust, with six research questions and concrete evaluations. The results show that NCDENow significantly outperforms baseline models, demonstrating its effectiveness in both normal and abnormal periods, such as during COVID-19.

3. Clear Presentation and Methodology:
The paper is generally well-written and easy to follow, with well-designed figures that effectively convey information. The detailed description of the algorithm and modeling approach further enhances the paper's clarity and accessibility. The inclusion of limitations and future studies at the end of the manuscript shows the authors' awareness of the study's boundaries and the potential for further research.

Areas for Improvement:

1. Baseline Comparisons:
Reviewers noted that the selection of baselines could be improved. The current baselines are somewhat outdated, and including more advanced models for forecasting and nowcasting would provide a more comprehensive comparison and validate the effectiveness of NCDENow more robustly.

2. Dataset Statistics and Generalization:
The paper would benefit from a more detailed introduction of dataset statistics, including the partitioning of training and testing datasets and the differences and similarities between the datasets used. Expanding the validation to include more countries and diverse conditions affected by irregular events would provide a better assessment of the model's robustness and generalizability.

3. Writing and Formatting:
Minor improvements in writing and formatting are needed. Polishing the language, correcting typos, and ensuring adherence to scientific writing standards would improve the overall presentation quality. Specific sections, such as captions for tables and figures, should be revised for clarity and completeness.

4. Broader Validation:
While the paper demonstrates the method's effectiveness in South Korea and the UK, expanding the validation to include more countries and diverse economic conditions would provide stronger evidence of the model's generalizability and robustness.