Script News Download

Avatar
Ulises Cune
  • updated

Hello Community,

I have created this Python script that downloads the news from the site www.forexfactory.com and saves it in the directory (C:\Users\your_name\AppData\Roaming\MetaQuotes\Terminal\Common\Files\News\FF\*. csv) that uses CP.

It does not insert the values of "revised" and "revised_time.", the rest of the values are the same as Andrey has in Google Drive.

You can read more about the News Filter

The news are downloaded from:

Image 3258

You need to download Python 3.x and install the plugins run "pip3 install beautifulsoup4" and all others that you do not have installed

Image 3260


CommunityPowerEA_News_Download.py

Replies 50
Avatar
0
Ulises Cune

Tell me how you are going to use the future news if you don't have the futures candles!

Avatar
0
th0mas51

I'm using future news schedule to stop trading, close positions or prevent to open new positions couple of hours or days before the event happens.

There's no need for future candles for that

Avatar
0
Jan Corsair
Quote from th0mas51

I'm using future news schedule to stop trading, close positions or prevent to open new positions couple of hours or days before the event happens.

There's no need for future candles for that

Yes and this is exactly what one should do if you are new to trading or if you have the reaction time of a human. BUT: CP EA is software and runs on hardware capable of much faster reaction. The volatility caused by news is caused by discrepancy between forecasted and real values of certain fundament indicators (not only, not always). I would like to see CP EA to develop more into using news volatility instead of avoiding it.

Avatar
0
th0mas51
Quote from Jan Corsair

Yes and this is exactly what one should do if you are new to trading or if you have the reaction time of a human. BUT: CP EA is software and runs on hardware capable of much faster reaction. The volatility caused by news is caused by discrepancy between forecasted and real values of certain fundament indicators (not only, not always). I would like to see CP EA to develop more into using news volatility instead of avoiding it.

I understand your point about having an Expert Advisor that is able to trade live news releases, but this it would require to query ForexFactory's website very often in order to refresh the "Real Values" in real time, as they are being released, and this would overload the ForexFactory website.

Also, this is not what this script is about. The main goal of this script is to download historical data for running backtests with CommunityPower.

Avatar
0
th0mas51

Quite frankly when I see how confusing the latest couple of comments have been, I regret ad ding custom changes to this script as it looks like it has made everybody confused.

So in order to keep everything simple, I have reverted my changes, and I'm posting a new version of the script with only minor changes to the last original version that was released by Ulises in December.

I also did a mistake in my previous fix about hours "10th-15h", as those events are not
in hours (10:00-15:00) but in days (between the 10th of the month and the
15th of the month).

In this new version of the script, the events that are using time "10th-15th" are now labelled as "00:00", meaning "All Day Event" instead of being labelled as "10:00" hour.

Here are the changes, in comparison to the original version that was posted by Ulises on December 2023-12-16:
- Updated the "all_day_events" logic so it can now use a regexp, to match more easily the patterns
- Added support for the time format "10th-15th" as "00:00", which means "All Day Event"
- Add "--force" parameter that allows to force re-downloading the news events of the last 2 months, in case you're not sure if your data is up-to-date or not
- Add a safety check to not re-download the files that have been downloaded less than 1 hour ago, to avoid overloading FF website

List of the recent improvements I have removed, that are not needed by CP:
- Removed the possibility to download the news using UTC timezone
- Removed the possibility to download next weeks' news

@Andrey, can you please delete my previous versions of the script, so people stop being confused, thanks !

HistoryLoad_v2_2024-01-17.py

Avatar
0
Ulises Cune

It looks like you still don't understand. To do backtesting you only have candles until today or yesterday, if you don't have candles, you can't do backtesting, therefore future news is of no use to you because you don't have candles.

CP downloads the news live from https://nfs.faireconomy.media/ff_calendar_thisweek.xml and not from parsing the page as the python script does.

Avatar
0
th0mas51
Quote from Ulises Cune

It looks like you still don't understand. To do backtesting you only have candles until today or yesterday, if you don't have candles, you can't do backtesting, therefore future news is of no use to you because you don't have candles.

CP downloads the news live from https://nfs.faireconomy.media/ff_calendar_thisweek.xml and not from parsing the page as the python script does.

Yes, I know that I can't backtest in the future.

Basically I did the improvements to the script because I was hoping I could use and maintain a single script for doing two different things at the same time :

The first usage of the script is of course the default usage, for downloading historical data for backtesting in CP.

And then I was hoping to use that script for my own personal Expert Advisor, I would schedule the script in Windows' Task Scheduler every day at 00:00 and it would generate the CSV file for this week and next week's calendar schedule. The goal of that is so that I don't have to code the logic of downloading the news live from FF inside my Expert Advisor, as it's a pain in the ass to parse the XML code and FF might change the XML code, and it's easier to read the content from the CSV file.

Anyway, as I saw that this was clearly confusing everybody, I removed the feature of downloading next week's calendar, as it's not needed in CP anyway, and I will maintain my own version of the script that I can use to provide news data by refreshing the CSV file every day.

Sorry for confusing everybody :)

Avatar
0
jay hu
Quote from th0mas51

Yes, I know that I can't backtest in the future.

Basically I did the improvements to the script because I was hoping I could use and maintain a single script for doing two different things at the same time :

The first usage of the script is of course the default usage, for downloading historical data for backtesting in CP.

And then I was hoping to use that script for my own personal Expert Advisor, I would schedule the script in Windows' Task Scheduler every day at 00:00 and it would generate the CSV file for this week and next week's calendar schedule. The goal of that is so that I don't have to code the logic of downloading the news live from FF inside my Expert Advisor, as it's a pain in the ass to parse the XML code and FF might change the XML code, and it's easier to read the content from the CSV file.

Anyway, as I saw that this was clearly confusing everybody, I removed the feature of downloading next week's calendar, as it's not needed in CP anyway, and I will maintain my own version of the script that I can use to provide news data by refreshing the CSV file every day.

Sorry for confusing everybody :)

Keep up the good work, th0mas51. Thank you for your help.
Avatar
1
Ulises Cune

I think this can be useful for everyone.

Script to determine whether your Broker follows the US, UK or AU daylight (DST) schedule.

The script scans the H1 chart bars at the expected dates for the daylight changes, and by comparing the changes in server (bar) times at these bars, the DST schedule of your broker can be determined.

DST_AU schedule:

DST_UK schedule:

DST_US schedule:

DST_NONE schedule:

  • the server does not forward its clock during summer.

Original:

https://www.mql5.com/en/code/48650


Image 5822


Scripts:

BrokerDaylightSchedule.ex5
BrokerDaylightSchedule.mq5

Avatar
0
jay hu
Quote from Ulises Cune

I think this can be useful for everyone.

Script to determine whether your Broker follows the US, UK or AU daylight (DST) schedule.

The script scans the H1 chart bars at the expected dates for the daylight changes, and by comparing the changes in server (bar) times at these bars, the DST schedule of your broker can be determined.

DST_AU schedule:

DST_UK schedule:

DST_US schedule:

DST_NONE schedule:

  • the server does not forward its clock during summer.

Original:

https://www.mql5.com/en/code/48650


Image 5822


Scripts:

BrokerDaylightSchedule.ex5
BrokerDaylightSchedule.mq5

Nice tool. Thank you for posting this!