ESG Ratings: Too Much ‘Garbage in’ ‘Garbage out’

By Kurt Lieberman, CEO, Magni Global Asset Management

November 2018

In 2016 the Global Sustainable Investment Alliance reported that sustainable investment assets reached $22.89 trillion — a 25% increase over 2014. While Europe accounts for over half the total assets, the US is among the fastest growing countries, with an increase of 33% in sustainable investing assets over the past two years. It’s no surprise that this growth has increased the number of ratings agencies dedicated to environmental, social and governance (ESG) research on companies around the world. The Global Initiative for Sustainability Ratings has identified over 600 ESG ratings agencies globally. This does not include firms that establish a framework for reporting and evaluation, such as the Global Reporting Initiative (GRI) standards or Sustainability Accounting Standards Board (SASB).

What is troubling is that each rating agency uses different methodologies, metrics, weightings, and in some cases, the ratings reflect their own agendas. Since there are different perspectives about what constitutes ESG, the resulting ratings must be considered somewhat subjective. In his paper, Ratings that Don’t Rate: The Subjective World of ESG Rating Agencies, Timothy Doyle of American Council for Capital Formation (ACCF) notes that “…each rating agency has a customized scoring method which evaluates different non-financial metrics and frequently disagree about the components of ESG. Determining which ESG topics and metrics to evaluate is not a straightforward exercise – certainly not when compared to traditional financial metrics”[1]. When comparing ESG ratings of companies in the S&P Global 1200 by two of the largest rating agencies, MSCI and Sustainalytics, sustainability analysis firm CSRHub found that the two ratings had a low correlation of .32[2]. Simply put, two firms looking at identical data, are coming up with totally different conclusions. A specific example of this is Bank of America (BA), which was scored by two of the largest rating agencies: Sustainalytics and RepRisk. RepRisk gave the bank an above-average score of AA, while Sustainalytics gave a below average score of 50. By contrast, when comparing credit ratings performed by Standard and Poor’s and Moody’s, there was a correlation of .90[3]. To our earlier point, the subjective nature of the variety of ESG ratings is concerning. Without a standardized format of analyzing audited information, the presumed result is ‘garbage in’.

Responding to ESG questionnaires: a source of pain for most

Companies are being overwhelmed by requests from ratings agencies requesting information to complete these ESG reports. In SustainAbility’s report, Rate the Raters 2018, one of the emerging themes is entitled “Crowded, Complex, Costly, Time Consuming”[4]. Requests are becoming more complex because there is no standardization of data being requested. It has become costly and time consuming not only because of the need to hire additional staff to complete these requests, but additionally some agencies charge the company a fee to file the information.

Many companies issue proprietary sustainability reports. In general, ESG rating systems reward companies with more disclosures. More recent research of over 4,000 sustainability reports from 2005–2009 finds a significant number of data omissions, unsubstantiated claims, and inaccurate figures[5]. Traditionally, this unaudited information is taken at face value. So, it is possible for companies with weak ESG practices, but robust disclosure, to receive an average or above average score when in fact they may have more overall ESG risk. These companies attempt to manipulate the system to get points in certain areas, often hiring third parties to help them improve scores. They have also employed proxy advisory firms to assist them in improving their environmental and social scores[6].

Inevitable biases and oversimplification

The ratings agencies appear to have inherent biases. Larger companies have more resources than smaller companies in preparing disclosures and tend to receive higher scores. Geographic bias exists as companies in Europe tend to have higher scores than North America because of the EU-required disclosures that are not required by the US. An example is American-made Tesla, which ranks below every European car manufacturer including Volkswagen[7]. There are also industry sector biases in rating companies based on industry instead of company-specific risks.

The ratings agencies’ goal is to score as many companies as possible and monetize their research. To accomplish this goal, most employ a “check the box” mentality; sending out questionnaires to companies for completion (as noted above), reading unaudited reports published by companies, and using opaque and proprietary methodologies to derive a score. Corporate sustainability professionals have expressed concern over the quality of information that is being published[8]. They are concerned about the oversimplification of data that is “…just boiled down to a number”. Companies may be rated on issues that are not material to their businesses nor their sustainability performance. The ratings agencies often become disengaged and do not update or upgrade scores even when positive changes have taken place at the company. According to a corporate stakeholder in the energy sector, “You can easily improve your scores without actually doing anything of value – for example, on one rating my score can go up by 10 points for employee training if I send a link to people, but am I really changing something in the business?”[9]

Hopefully, these ratings will begin to converge. Standards will eventually be established, and audits of company disclosures will be more common. The two key issues seem to be: (1) how do we accelerate the process so that the data is more usable and (2) what should we measure? By measuring what is truly important, the convergence of scoring should occur faster; otherwise we will continue to observe ‘garbage out’ from the rating agencies.

Actual behavior should be front and center

What is missing from existing ESG ratings is the research process and measurement system for behavior. The behavior in a company reveals what is truly important. Companies may have rules and regulations and publicly state their beliefs and intentions, but that doesn’t always translate into what is actually happening in a company. In the movie Casablanca, gambling was forbidden in the city of Casablanca, but it was not enforced. When Captain Renault of the Vichy police shut down Rick’s Café, he exclaimed he was “shocked, shocked!” to find out that gambling was going on, at the same time he was given his cut of the winnings from a croupier.

Understanding behavior shines a light on the actual quality of corporate governance. Good corporate governance is based on honesty and transparency and should be assessed through its stakeholder relationships. This includes the CEO, board of directors, employees and shareholders, as well as how the company interacts with its customers, suppliers, competitors and the community in which it operates. The measurement system should encompass how a company reacts with all these constituents.

Evidence suggests that governance plays a key role in ESG portfolios. The connection between good governance and good environmental and social performance at a country level has already been shown[10]. Sustainability at a company level requires good governance[11]. In addition, “The results are unsurprising in regards to good governance being a critical driver of investment performance,” says Devan Kaloo, Head of Equities at Aberdeen Asset Management. He also notes that, “Aberdeen believes companies adopting best practices in corporate governance will be more successful in their core activities and deliver enhanced returns to shareholders.” Deutsche Bank’s 2012 whitepaper, Sustainable Investing: Establishing Long-Term Value and Performance, documents that better ESG performance came from positive screening and governance had the strongest influence on financial performance[12]. Nearly 90% of financial services decision-makers considered effective governance to be a critical driver of investment performance, according to data in the paper.

The traditional “check the box” analysis of ESG investing is nothing more than negative screening. The agencies are looking for disclosures and proclamations by companies that are unaudited and then basing their decisions on that data. They are looking for anything that fits in their box (either “yes” or “no”) that a company says it is or is not doing. Positive screening means evaluating each question to find out if the company is really doing what it says it is doing. The key is understanding the actual behavior of the company rather than making a quick judgement and moving on to the next question.

ESG company ratings must become more reliable and believable

ESG investing will likely continue to grow rapidly, as will the number of ratings agencies conducting the so-called research as the basis for stock selection in portfolios. It is becoming widely suspected that the research is biased, unaudited and subjective. If there is too much ‘garbage in,’ then all we are going to get is more ‘garbage out.’ Investors should have access to a more rigorous and insightful methodology that incorporates an objective and repeatable way to measure the actual behavior of a company. That would be a sustainable rating.



[2] Data sourced from subset of 4,150 Sustainalytics companies and of 150 RepRisk companies.










[12] “Sustainable Investing: Establishing Long-Term Value and Performance”, DB Climate Change Advisors of Deutsche Bank Group, June 2012, page 8