July 14, 2014

Premium Data from Norgate Investor Services Review

THIS REVIEW IS OLD. READ THE NEW REVIEW: Norgate Data Review

Old Review below.

I am frequently asked what data provider I use. A year ago my data provider was CSI Data. Then I heard about Premium Data from Norgate Investor Services and the one feature that enticed me to look at them closer: historical S&P500 index constituent data. At that time, I was maintaining the data by hand. Each month I would have to determine which stocks had been added or deleted from the index. I would need to look for name changes in the current and historical list. Not a hard task but time consuming and easy to make mistakes. The thought of not having to do this was very enticing.

This review will focus on US Stocks and AmiBroker integration. Premium data has data for the Australian and Singapore markets, integrate with multiple other platforms and have forex and futures data. For more information go to Premium Data from Norgate Investor Services.

Disclosures: I have paid for my data. The links on this page are affiliate links.

Features

While investigating Premium Data, these were the features I focused on:

  • Data back to at least 1995
  • Delisted stock data
  • Data quality
  • The data can be adjusted for dividends and one time capital gains
  • A watch list of the S&P500 index constituents
  • A simple way of determining if a stock was in the S&P500 on a particular date
  • Data integration with AmiBroker
  • Good customer service especially when dealing with bad data
  • Reasonably priced compared to CSI data

 

We will look at each area in more detail with each area receiving a letter grade.

Data back to at least 1995

Premium Data has US stock data back to 1950 which is much further back than I would ever test. One can purchase data back to either 1950 or 1985. I choose 1985.

Grade: A

Delisted stock data

This is an additional package added to the basic package. Coming from CSI Data, there is a difference on how stock symbols are treated when delisted. CSI data symbols are in the format of XYZ-123. Where, XYZ is the current symbol ticker and 123 is a unique number given to every company that does not change. For example Sun Microsystems symbol was SUNW. The symbol in CSI was SUNW-5968. Then they changed their symbol to JAVA. The new symbol in CSI became JAVA-5968. I could use this number to keep track of stocks and not have to worry about symbol changes.

Premium Data does things differently. Any currently traded stock has just the symbol. When a stock gets delisted, then they append the year and month. For example Sun Microsystems symbols is JAVA-201001. One nice benefit of this is that in your trade list it is easy to spot stocks that are delisted.

Grade: A

Data Quality

This proved to be the hardest part of my evaluation. The first tests were running several of my strategies through both Premium Data and CSI. The results were close but not super close. This gave me confidence that both data sets were similar. The problem is how does one say which data source is correct without a third data source to compare against.

I could compare about a dozen symbols with Bloomberg’s data. Eight symbols agreed between CSI and Bloomberg. Four symbols agreed with Premium Data. Of the eight symbols were Premium Data did not agree there were a couple of obvious data errors, which I reported to Norgate Investor Services.

Grade: A

The data can be adjusted for dividends and one time capital gains

There seems to be some misconception that Premium Data Services cannot do this. I have had several people write saying they will not use the data because it cannot. The issue is that this feature is not turned on by default. In their support page, they walk you through on how to do this.

I compared Premium Data against CSI data and got the same results for ten dividend yielding stocks.

Grade: A

A watch list of the S&P500 index constituents

When investigating this feature, I thought I had died and gone to data heaven. Not only did Premium Data have a watch list for S&P500 historical constituents, they also have the data for the following indexes: DJIA, NASDAQ 100, Russell 1000, Russell 2000, Russell 3000, S&P 100, S&P 400, and S&P 600. In the past I had made a watch list of my trading universe. This now could save me a step and I could always use the Russell 3000. What a time saver.

Comparing my S&P500 historical list to Premium Data’s they were very similar. Nearly perfect match for the last 5 years.

One important note on this, the historical constituent data is currently a beta program for AmiBroker users. You need to ask to get into this program. I do not know if this data is available to other platforms.

Grade: B (A+++ for multiple index data, C because it is a beta program)

A simple way of determining if a stock was in the S&P500 on a particular date

Determining if a stock is currently in the index could not be simpler. Here is sample AmiBroker code.

inIndex = IsIndexConstituent(“$SPX”); // test for in SP500

inIndex = IsIndexConstituent(“$RUA”); // test for in R3000

Grade: A

Data integration with AmiBroker

Premium Data integrates directly into AmiBroker. After going through Premium Data’s setup instructions, which are very good, you need only to run their data update program and your data is updated in AmiBroker. Nice and simple. Much easier than what I was doing before.

One thing to note, which does not affect me, is that there are no intraday updates. Meaning if you run the update program during market hours, one still is getting the previous day’s close. Not a 20 minute delay snapshot of the current day. This may be important for people who like to run their scans right before the close.

Grade: A

Good customer service

I am always on the lookout for bad data. Sometimes it is one day with a bad price. Other times, it is missing data for several days. Either way, it is important that my data provider quickly look into the issue and get back to me. Data is my life blood.

While comparing against CSI Data, I found some obvious data errors. I sent these errors to Norgate Investor Services. Within 24 hours they had responded that it was an error and they were fixing it. The fix appearing within 48 hours which is great turnaround time.

They have also been very good about responding to any technical support emails. They are located in Australia, so emails are answered at night for those of us in the United States.

Grade: A

Reasonably priced compared to CSI data

Given the index data and the tight AmiBroker integration, I would pay a little bit more. Here are the costs:

Daily data: $297/year

Currently listed US Data History back to 1985: $75 (single time fee)

Delisted US Data History back to 1985: $232 (single time fee)

First year cost: $604

Subsequent years cost: $297

I am paying $680 per year for CSI Data. Another benefit is that I am using a different data provider for my daily signal generation which is costing $300 per year. I will be able to drop them. After the first year, may data costs will go from $980 to $297, this will be a substantial savings.

Grade: A

Misc.

Premium Data allows the creation and linking of multiple AmiBroker databases. This is helpful because I like keeping three databases: one with 2 years of data for daily scans, one with 11 years of data for most of my back testing and the third with 25 years of data when I am curious about results in the 1990’s.

Premium Data can be installed on two different computers at the same time.

Here we have a screenshot of the update program

140714a

Conclusion

I have been using Premium Data exclusively for my testing for the last three months and have been very happy. The only minor negative is that the constituent data program is in beta. The last time I asked there was no date on when it would come out of beta.

Even without the constituent data, I would recommend Premium Data. Having the historical index constituent data makes me highly recommend Premium Data.

Click the logo below to get more information.

140714b

Visited 42 times, 1 visit(s) today

Click Here to Leave a Comment Below

David - July 14, 2014 Reply

I switched to Norgate a couple years ago, following 12 loyal years with a different data provider. I’ve also been VERY happy with them. A must have for any professional quantitative investor. Customer service is incredibly responsive in my experience.

Ronen - July 15, 2014 Reply

Hi Cesar,

I was on the verge of of subscribing to Norgate on the weekend, and your review will now help me with the decision to go forward. I agree with your conclusion, that it is very difficult to say from multiple data sources, which one is the correct one. Even with a third comparison it could still be inconclusive if that third one gives a completely different result from the other two.

The S&P constituents is very interesting and might be a first with data vendors. I couldn’t imagine doing the work my self. I would just be too labor intensive. Hopefully this will be part of their existing delisted data package and not an add-on cost once it gets out of the Alpha stage.

I’ve got only one question. Does the data integrate with 64bit Amibroker?

MachineGhost - July 15, 2014 Reply

Ummm, you gave Data Quality an A when only 4 of 12 stocks match Bloomberg. Come on! That deserves a C at best.

Support has indicated that they adhere to standard industry practice of not adjusting for ordinary dividends in line with the major index providers. So where is this option to turn on back-adjusting stock for ordinary divdends so there’s not little gaps all over the charts?

The Data Integration is not a native plugin so it requires a manual update to refresh the AmiBroker database which is done by running a provided script which is prone to screwing up. This does not deserve an A either.

Re cost, there is no monthly plan as is standard with other data providers. You must pay 6 or 12 months upfront. Again, not worthy of an A. Maybe you think you’re saving money, but CSI’s historical stock data has free via Yahoo for almost two decades now.

    Cesar Alvarez - July 15, 2014 Reply

    I was not clear. After comparing over 1000 stocks, there were about 12 stocks that stood out. Even CSI had obvious errors. There was a time that I was finding and sending monthly errors to CSI. I would like my data provider to be perfect but do not expect that.

    As to dividend adjustment. In AmiBroker go to File/Database Settings/Configure. In this dialog you can select if the data is adjusted for ‘Capital events’ and/or ‘Special cash dividends’ and/or ‘Ordinary dividends.’ As I stated I compared CSI Data and Premium data on dividend stocks and got the same numbers.

    As to integrating with AmiBroker, maybe we have different meaning. Under ‘File/Database Settings’ ‘Premium data plug-in’ is an option under Data source. There is no plug in for CSI Data. I also use TeleChart, which has a plug-in. In order to update TeleChart data, I need to run TeleChart and update the data there first. TeleChart and Premium data behave the same. Boot them up and click update. I have not had any issues with my updates.

    You are correct that there is no monthly plan. I have always paid yearly with CSI Data and TeleChart so I guess this does not bother me. You are right, if you want a monthly plan then this is not for you. I should have mentioned that in the initial review.

    One can get free CSI data from Yahoo but it does not include delisted stocks and historical constituents. If one is not looking for that, then the Yahoo data would be fine. Premium data is not for everyone. Since I make my living from the data, I want to make sure there is someone there to fix issues when I find them and is delivering high quality data. Premium Data meets my needs very well at a great price.

      MachineGhost - July 16, 2014 Reply

      OK, I think you need to make clear in the review that you are using the beta version which has native AmiBroker plugin support. AFAIK, this beta is not generally available to subscribers unless that has changed recently. I see no news announcement to that effect. So until the beta is no longer in beta, we will not be able to replicate your experience of not having ordinary dividend gaps all over the place and not having to use unstable integration scripts to deal with ticker changes, delistings, etc..

      Frankly, it’s the gaps that make Norgate unusuable for backtesting over CSI/Yahoo. And to add insult to injury, they want 6-12 months paid upfront?

      Yes, I’m greatly annoyed because it seems like the classic case here of engineers/programmers taking over the company and who have very little business or marketing acumen. If support doesn’t seem fit to publically mention how to access the “Alpha Test” database, what use are they?

Rob Angerer - July 15, 2014 Reply

Thanks for the very clear review and recommendation. I signed up with Norgate today based on your and Matt R.’s recommendation. The historical index constituent capability is a tremendous step forward.

Nick Radge - July 15, 2014 Reply

Great review. I’ve been with Norgate for many years and have been using the Alpha Test database for well over the last year. All trading and testing is done with this and it works a treat. Support is second to none.

    MachineGhost - July 16, 2014 Reply

    So pray tell us, how do we access this “Alpha Test” database?

    MachineGhost - July 16, 2014 Reply

    And even better, do trial subscribers have access to it? Because if not, this sore fact needs to be mentioned in the review as well.

      Cesar Alvarez - July 16, 2014 Reply

      To get access to the Alpha/Beta program, one simply needs to email support that you want access. I do not know if it is part of the free trial. I jumped straight to the Alpha/Beta program. I have made a change to original post stating that the review is the Alpha/Beta program. Sorry for the confusion.

        Rob Angerer - July 16, 2014 Reply

        Ditto. I signed up for a year in the morning, sent them an email asking for Alpha(Beta) data manager program after I got the various confirmation emails, and they sent me instructions for the Alpha(Beta) data manager within hours… downloaded and using with no problems so far.

        Don’t know about trial subscribers.

          Cesar Alvarez - July 16, 2014 Reply

          Good to hear that it has worked out for you.

Dean - July 16, 2014 Reply

Could you post the link to Norgates support page that explains how to make the adjustments for dividends and capital gains?

Many thanks for a great article.

    Cesar Alvarez - July 16, 2014 Reply

    I can’t find the web page but here are the instructions:
    In AmiBroker go to File/Database Settings/Configure. In this dialog you can select if the data is adjusted for ‘Capital events’ and/or ‘Special cash dividends’ and/or ‘Ordinary dividends.’ As I stated I compared CSI Data and Premium data on dividend stocks and got the same numbers.

Rob - August 8, 2014 Reply

Cesar,

Is there a speed increase in the daily EOD data update with Norgate as compared to Amiquote ?
My daily EOD universe has 2400 tickers and this takes approximately 3 hours with Amiquote.
Would Norgate or TC2000 be able to get the data faster?

Rob

    Cesar Alvarez - August 8, 2014 Reply

    I cannot compare to AmiQuote because I do not use it. For TeleChart, my first daily scan takes about 3-5 minutes. My next ones each take about 30 seconds. The way to make scan go much faster is to make a special scanning database that contains the minimum number of bars you need. In my case, this scanning database only has 300 bars.

Ola - August 30, 2014 Reply

Thanks for a good review. I’m just about ready to ditch EODdata.com after having lots of missing quotes and a number of unanswered support requests.
Even with the help of Amibroker’s Database purify tool it is a time consuming process to fix the holes I can teill you!
A friend of mine is a Premium data subscriber and very happy, so I’ll go with them definitely.
I’d be very interested in learning more about backtesting without survival bias using Premium Data. Maybe something for your future posts?
Thanks again for valuable info.

Gary - September 4, 2014 Reply

Cesar,

Premium Data provides delisted and listed stock data in separate packages. There is no way of knowing which delisted stocks existed on which index.

So how do I run a backtest on the S&P 500 watch list and account for survivorship bias accurately?

It seems to me the only way is to run back tests on the Russell 3000 and include all delisted securities.

Many thanks for your advice.

    Cesar Alvarez - September 4, 2014 Reply

    In their Alpha program, which you need to ask to join, they have the historical constituent data. Then you use this function, IsIndexConstituent(“$SPX”), to tell when a stock is in the index. For example:

    Buy = RSI(2) <1 and IsIndexConstituent("$SPX");

JP - November 11, 2014 Reply

Thanks, Cesar.

I decided to purchase the Premium Data stock data based on this review. My only criticism is that your first year calculation for cost is $75 off. Norgate requires purchase of both the currently listed database and the unlisted database. Therefore, the first year cost is $604 ( 297 + 75 + 232 ) rather than $529. Otherwise, great review.

    Cesar Alvarez - November 12, 2014 Reply

    Thank you for pointing this out. I probably did a dumb math mistake and did not add the $75 fee. I have fixed it.

Ern - December 22, 2014 Reply

Hi Cesar,
re:
“inIndex = IsIndexConstituent(“$SPX”); // test for in SP500”

Is this Amibroker function “IsIndexConstituent” a new function provided by the premium data DB plug-in?

Also a question on the quality of the indices historical constituent data – do you know if it been verified against research databases such as Compustat? and/or alternatively have you verified it prior to 2005?

Also is fundamental data provided and accessible via Amibroker such as Market Cap? .. and
would you know if the fundamental data is point-in-time based or current?

Thanks,
Ern.

    Cesar Alvarez - December 22, 2014 Reply

    Ern,

    The ‘IsIndexConstituent’ function is provided by the plugin. It is only available under the Alpha program.

    I compared the S&P500 data to a source I had (not Compustat) and it was a very close match. The farther you go back the less I trust any database.

    I have no idea about the fundamental data. Please share if you find the answer to this one.

    Cesar

Jason - April 22, 2015 Reply

Hello,

Just a bit of insider info as I’ve used both services – CSI and Norgate – and am very familiar with how their data is obtained. I find this article to be somewhat ironic.

Ironic because Norgate actually sources their data from CSI! As does Yahoo, and many other large financial companies who would prefer not to be mentioned. If you don’t believe me, try both services at once, and compare the times that the final data update is released. Norgate will always be right behind!

Also note that CSI boasts a far wider range of markets/exchanges and provides prompt customer support. Norgate’s product offering is like a small subset of CSI. See for yourself:

http://www.premiumdata.net/products/datatools/futures-coverage.php
http://www.csidata.com/?page_id=10

Yes, Norgate is cheaper. But like anything else, you get what you pay for. CSI uses a variety of sources and are constantly comparing and disputing suspect data. Who do you think is going to be accountable when a bad data point comes through? Norgate will be waiting on CSI for that corection….

Jason

    Cesar Alvarez - April 22, 2015 Reply

    Jason,

    I have had (and still do) CSI data for 10 years. I am familiar that Yahoo and other data providers get there data from CSI. I did not suspect that Norgate gets there data from CSI. I will have to asked them. When I first decided to look into Norgate, I and two other researchers did lots of comparison between CSI & Norgate. They did not always match. And when they did not, it was not always one who was wrong when we compared it to Bloomberg.

    Yes CSI offers more markets but that does not matter for me since all I care about is US stocks. Where Norgate exceeds CSI is the fact that they have historical constituent data. At Connors Research we had kept that data up to date by hand and had it back to 2001. It was a royal pain to do. Now I get that for ‘free’ from Norgate data which is a huge timesaver to me.

    Another potential issue for new subscribers to CSI that about two years ago stopped offering delisted stocks to individuals. Again that is important for me to have.

    At the end of the day no provider is perfect. One needs to understand the trade offs and go from there.

    Cesar

Rob - June 6, 2015 Reply

Cesar,

Can Norgate or CSI data be updated automatically say from Ami Broker?

I currently use TC2000 and the data update is a separate action from running my scripts in Ami Broker and I would like to automate as much as I can.

Rob

Cesar Alvarez - June 6, 2015 Reply

@Rob – For both CSI & Norgate you need to update outside of AB. Just like TeleChart.

Robert - June 7, 2015 Reply

Hi,

very interesting discussion going on following your post.
A bit out of topic, any thoughts on IQFeed historical data for US stocks?

Cheers,

Robert

    Cesar Alvarez - June 7, 2015 Reply

    I have never used them so I have no comments on it. Sorry

Phil - December 10, 2015 Reply

Norgate seems a reasonably priced option for US stock data, but I don’t fully understand the implications of these statements (from their website):

“The Historical Database is fully (and consistently) adjusted for capital events (splits, capital returns, demergers etc) which allows trading strategies to be back-tested on a constant capital basis (there is no adjustment for ordinary cash dividends)”

“Why aren’t US Stocks adjusted for ordinary dividends?
Our stocks data is adjusted for all capital-related corporate actions – splits, reverse splits, capital returns, special dividends, stock dividends, demergers and spin-offs. The methodology we use is the same as that used by index providers (such as Standard & Poors) for benchmark indexes such as the S&P 500, NASDAQ 100, Russell 2000 etc. These indexes are not adjusted for ordinary dividends. By using the same methodology as the index providers, we keep our stocks data “in synch” with the indexes.”

Why is it valid in backtesting to ignore cash dividens (ordinary dividends?)? It seems to me any dividend is valuable…

    Cesar Alvarez - December 10, 2015 Reply

    Phil, I am not sure how you come to the conclusion that “it is valid in backtesting to ignore cash dividends.” Norgate does not state this. They are stating the reason tehy do not do it is because that is how everyone else does it. If you go to Yahoo and look at any stock chart, the prices are not adjusted for dividends. I cover this issue in more detail on this post

Phil - December 10, 2015 Reply

Sorry, I didn’t read the entire comment section. I see you can adjust the prices for cash dividends in AmiBroker, any idea how to do this without AmiBroker? I’m just using their Premium Data tool to manage data.

    Cesar Alvarez - December 10, 2015 Reply

    Sorry but I don’t. I would email Premium Data. Their support is really great.

David - April 8, 2017 Reply

datafet.com also provide Real-Time stock data for Metastock, AmiBroker & Metatrader at very good price

    Cesar Alvarez - April 8, 2017 Reply

    But for my needs they do not appear to have deliisted stock data.

Leave a Reply: