Debugging Report Residual P&L Discrepancy – Has Anyone Investigated This?

[Spacious08]

Hi everyone,

In the daily PnL report, I’ve noticed persistent residual P&L discrepancies between the P&L calculated from my actual positions and the P&L reported by my broker account. Occasionally, these residuals are small, but 1-2 times per week, they exceed 1%, which should not happen since I’m collecting all the required price data just before the calculation. Here’s my current setup and additional context:

Current Setup

1. Data Collection:

• Instrument prices are collected daily at 18:00 UTC-5 when the markets are "closed."
• My broker account balance is updated every 2 hours, including a timestamp at 18:00 UTC-5 to align with instrument prices.

2. Report Generation:

• The reporting process runs at 19:45 UTC-5, after all calculations are complete before 20:00 UTC-5 when the Asian markets start opening.

Additional Notes
I’ve added Forex impact from my cash balance into the report to determine whether it could explain these residuals. However, so far, these Forex variations have never been the cause of residual P&L values exceeding 1%.

Question
Has anyone debugged similar issues in depth? Specifically:

• What could cause significant residual P&L discrepancies like this?

From my perspective, this suggests a bug or inconsistency somewhere in the calculation pipeline, but I haven’t pinpointed it yet. I plan to investigate further this week, but any insights or advice from others who’ve encountered and resolved similar issues would be appreciated.

Thanks in advance for your help!

[tgibson11]

Daily P&L is based on the futures closing prices.

Not all markets close at the same time, not even all US markets. Of course this would be especially relevant if you're trading on exchanges on multiple continents.

"Closing" is more or less a formality in most markets. Trading continues after hours, and will be reflected in your IB account balance. Many markets trade very close to 24 hours Mon-Fri. For example, S&P500 e-minis close at 16:00 ET, and re-open at 17:00 ET.

Major news is often announced shortly after the equity market close. Earnings announcements that would affect equities, but other things too. All this will be reflected in the IB account balance, but not (yet) in the daily futures price data.

It's certainly possible that there could be bugs in the P&L calculations, but I don't think what you describe is necessarily evidence of it.

[Spacious08]

Thank you for the response and insights. I wanted to share some additional findings based on my debugging efforts:

GMT Offset and Closing Price Issues:

I discovered that people using GMT_offset_hours = -5 (like myself) can encounter difficulties collecting end-of-day closing prices.
Specifically, because I’m collecting daily and closing instrument prices at 18:00 UTC-5 and the notional closing time in PST is actually 23:00. As a result, the closing price is considered has a price in the futur and is rejected.

Impact of ignore_future_prices Configuration:

I initially attempted to change the ignore_future_prices option in my configuration to False. This allowed closing prices to be processed, but it had unintended consequences, such as capturing day-ahead closing times for Asian markets already open.
Rob’s documentation on this option clarifies:
If ignore_future_prices is True (default: True), prices with timestamps in the future are ignored. This ensures we don’t prematurely include Asian market closing prices in the dataset.

To avoid these complications, I reverted to the default configuration with ignore_future_prices = True.

Potential Solution: Adjusting NOTIONAL_CLOSING_TIME_AS_PD_OFFSET:

One alternative solution I considered was modifying the NOTIONAL_CLOSING_TIME_AS_PD_OFFSET to 18:00:00. However, this would require retroactively adjusting all historical data (contract, multiple, and adjusted prices) to shift their closing times from 23:00:00 to 18:00:00.
Given the uncertainty and effort involved, I did not use this approach.

Conclusion:
The code responsible for fetching instrument prices (e.g., for forecast calculations), relies on hourly and daily closing prices, resampling them with .resample("1B").sum().
This means that even without the 23:00:00 closing time, my daily data are good enough to calculate forecasts

At this point, I don’t plan to change anything in my setup.
The residual P&L discrepancies I’ve observed are likely due to the reasons outlined by @tgibson11.