We introduce ForexDiff, an innovative deep learning framework for probabilistic forecasting and portfolio optimization in the foreign exchange (forex) market. Our framework consists of three key components: deterministic forecasting, probabilistic forecasting, and portfolio optimization, forming an end-to-end solution for quantitative trading in the forex market. We evaluate our framework on three major currency pairs (AUD/USD, EUR/USD and GBP/USD). The experimental results reveal that our framework can achieve consistent performance across different currency pairs. The average five-day cumulative return on AUD/USD, EUR/USD, and GBP/USD are 0.3\%, 0.2\%, and 0.5\%, respectively, highlighting the ability of our framework to transform uncertainty estimation into effective quantitative trading.