B-school rankings: Setting the expectations

In my previous column, I underlined the importance of ranking professional educational institutes.

During the ranking exercise, a comparison of institutes is carried out, within a certain geography. on several parameters of relevance and their sub-parameters. The critical issue is: whose inputs to take for these comparisons? Common sense says only those who have availed the services of B-school programmes have the knowhow on what should be compared and what should not.

If one strictly goes by the "experience" factor, it is impossible to find a person who has experienced all or several B-schools. A person generally experiences business management education at one of the many B-schools. However, during that experience he/ she develops enough ideas and insights on what makes a B-school better than another by way of discussions during admission processes, college-fests, inter-college competitions, during summer internships and at the time of placement. He/ she can say with certain amount of confidence that institute X is better than institute Y among a set of institutes.

It is important to compare a service or product from the perspectives of demand as well as supply. The supply side should be given an opportunity to present its credentials, while the demand side (users) would be best to judge these service providers based on their experience of the outcome. Hence BT-MDRA has devised the ranking based on three key components:

A. Objective data:
Business schools are encouraged to participate in the ranking process by putting up their case. They showcase their offerings, their achievements, best practices and other such information through an objective participation form which ensures comparable data on each measurable parameter or sub-parameter. This forms the "supply side" of the entire value chain.

B. Experience-based Perceptual data:
A large set of stakeholders use the rankings. However, not all fulfil the criteria of having used the services. Based on the feedback of those who can compare B-schools, an experience-based perceptual survey is conducted among:

a. Academia - deans/ directors (peer ranking)
b. Students - final year students who have "experienced" the offerings (user ranking)
c. Corporates - the market perspective based on the evaluation of final product (market ranking)
d. Alumni - those who have experienced the outcome of the offerings (user ranking)
 
C. Experiential data:
The assessment of the institute in terms of impact on their career growth, which is the primary purpose of enrolling into a business management course. (Career progression ranking). (This is for the first time that such experiential data has been availed of anywhere in the world).

The BT-MDRA methodology does not limit itself to getting the data from right set of respondents, but also ensures the accuracy of data. The validation and audit of the data is done religiously.

With this methodology wherein the inputs are taken from demand and supply - both sides, based on experience, that is further validated through a systematic process; it aims to make itself the most credible ranking

However, like any exercise it has certain limitations, such as:

a. Measurement: Not every parameter can be measured. As someone truly said, "Beauty lies in the beholder's eyes", but beauty contests are held across the globe every year. Similarly, even if some parameters cannot be measured, a comparison is important based on measurable differences. However, every year we aim to make more attributes measurable and try to use them too without affecting the comparability of rankings year-on-year.

b. Participation: To have a fair comparison, the honest participation of the institutes is important. Not all are motivated to participate due to various reasons - some obvious some non-obvious. In the absence of participation by certain B-schools due to whatsoever reasons, the entire ranking table has a chance to go topsy-turvy.

c. Relativity: Like any competition or comparison, ranking is a relative exercise. The evaluation and scoring are relative to each other among the participating institutes. Therefore by non-participation of some B-schools, the entire community loses out in two ways - i) by not knowing about the non-participated B-school and ii) relative rankings of other B-schools going up or down.

Despite these limitations, the BT-MDRA ranking approach tries to be most comprehensive in terms of taking the facts and viewpoints based on experience. It considers the inputs of widest set of stakeholders from the supply and demand sides of the B-school spectrum. It also does not mix up the 1-year programs with 2-year programs as is done by several rankings.

However, there is no single ranking either in India or globally that has been left out of criticism. Most of the rankings published by Indian magazines take perception-based data. Even if some rankings use objective data, they lack the rigour in validation of data.

Talking about the rankings by world's reputed publications; each one of them has its own merits and demerits. The six most widely referred global rankings are briefly discussed below:

1. The Economist: 130 institutes are invited to participate in rankings. 80per cent weight to objective data and 20per cent weight to student and alumni ratings. Further weights to objective data based on 4 factors - new career opportunities (35per cent); personal development/educational experience (35per cent); increasing salary (20per cent); and the potential to network (10per cent).

50-30-20 Rule: The weighted average of the previous year's data is also considered for the current ranking although the percentage decreases with year.

Pros: It features regional ranking. The 50-30-20 Rule makes in reducing volatility.
Cons: Policies on penalty for data misrepresentation are not clear which might encourage Business Schools to inflate their data. Also top 100 B-schools are ranked based on 130 invitations only. Market (recruiters) view is ignored.

2. Forbes: It is based on return-on-investment concept. Forbes sends out surveys to MBA alumni five years after graduation and asks their current salary and bonus. The magazine then subtracts out two years of forgone pre-MBA pay, as well as tuition and fees-minus scholarship aid. The actual Forbes ranking is based on the net cumulative amount alumni of a school have earned. The magazine adjusts the median five-year MBA gain for cost of living expenses and discounts gains using a rate tied to money market yields.

Pros: Much more effective for price sensitive students/stakeholders
Cons: Dependence on alumni for data. One-dimensional (only alumni perspective)

3. Financial Times (FT): Alumni responses on eight key criteria together (salary data contributing highest) contribute 59per cent weight. Eleven criteria (measuring diversity and international reach) are calculated from school data, accounting for 31per cent of total weight. A research rank accounting for 10per cent weight based on number of articles published by full-time faculty in certain journals.

Pros: Multi dimensional- includes "value for money" perspective as well. When taking alumni ratings, considers 3-year ratings where available.
Cons: The weighted salary calculation has flaws which sometimes over or underestimates salary grossly. Also, it gets very high weight. Critics point that salary may be low even when the institute is providing best opportunities and exposure.

4. Bloomberg Business Week: Judges B-schools on how well they serve their two main constituencies: students and corporate recruiters. Based on the web-based survey rating responses of students and recruiters. Students give ratings on 10-point scale on teaching quality, career services, and other aspects. Recruiters rate the programs based on perceived quality of grads and company's experience with MBAs.

Pros: Uses three most recent student surveys to ensure that short term issues do not skew results. Focus on customer satisfaction.
Cons: Rating scores by MBA aspirants may not be true. Completely ignores B-school's perspectives.

5. QS Global: The QS Global 200 Business Schools survey captures the preferred set of business schools each responding employer wishes to recruit. Each time a school is selected by an employer, it receives one vote and the total number of votes is referred to as the 'total unprompted votes.' Employers are then asked to identify the schools they regard as attractive for the purpose of hiring MBA graduates Each time a school is voted, it is given one vote, referred to as the 'total prompted votes'. The prompted and unprompted votes are added together to create the 'total employer votes'. Ranking is done according to 'index of employer votes'.

Pros: Ranks according to employers' perspective which helps in analyze the current demands and trends.
Cons: The ranking is one-dimensional as it is solely based on the opinions of the employers and excludes other stakeholders such as students, alumni and the perspectives of B-schools.

6. US News: It uses a wide variety of data sources. Quality assessment (weight 0.40) is based on peer assessment score (rating by other B-schools deans and directors, weight 0.25) and recruiter assessment score (survey among corporate recruiters, weight 0.15). Placement success (weight 0.35) includes mean starting salary and employment rates of grads. Student selectivity (weight 0.25) includes mean GMAT and GRE scores, mean undergraduate GPA and acceptance rate.

Pros: Multi-Dimensional approach based on quality assessment, placement success and student selectivity.
Cons: Does not seek ratings of students or alumni; ranks only US-based management programs.

Therefore each of the rankings methodology has their own advantages and shortcomings. Nonetheless, rankings are quite useful in several aspects of decision making. We at BT-MDRA try to evolve our approach year-on-year by incorporating new measurable elements and learning from our past mistakes to make it most credible and most sought-after rankings. By far, it is also most transparent in terms of publishing the details.

(The author is Executive Director, MDRA)