When it comes to education few countries in the world are as close to India in terms of competition; well on second thoughts China would be one. Considering the population of either of these countries it's not surprising that competition to get entry into one of the top institutions would be so cut throat. After all the students know getting entry into a good college is the assured way to secure a good future for themselves and their families. Thousands of students appear for entrance examinations every year at the culmination of a journey which involves a lot of sacrifices in both monetary and emotional terms.
Thus it is important that tests organized for admission to these institutes are unbiased to the maximum extent possible; any bias could provide an undue advantage to one group of students over the other. In India there are a number of entrance examinations conducted some at an all India level and some at State levels. And for each of them we have quite often heard how one section of students is at disadvantage over the other. For example, it is widely known that students pursuing plus 2 in CBSE board are more at advantage (because of the syllabus content and coverage) than the students pursuing a State board for plus 2 study. And hence to counter this the State level competitive examinations are tuned to the State board syllabus.
Now this is the case for entrance examinations for engineering and medical colleges across the country. It is even more complicated for admission to Science and Humanities courses in colleges.
Because of the wide variety of Secondary and Higher Secondary education boards across the country, there is a wide variance in the marking pattern and the marks obtained by the students. Some of the colleges ignore such a ground reality and consider the marks obtained by the students in Plus 2 as-is on their face value. While others have their own algorithm to normalize the marks obtained by students in different board examinations. And then there are some other colleges/universities who further tax the students by asking them to their specific entrance examinations.
A case in point is that of Jadavpur University (JU), Kolkata which few weeks back went through a period of turmoil on this issue of admission test. The admission test was initially scrapped but then under pressure from students the tests were restored. However now another set of controversy has struck the University with a section of aspiring students having raised the issue of marks they got in the university conducted admission tests; they scored more than 90% in board examinations but got single digit marks in the JU entrance test.
So the question now is which examination to rely upon in order to judge the ability of the aspirants and their readiness for a University course. Some say it's the board examination and some others say it is the college / course specific entrance tests. And so the middle path is mostly followed; considering both Plus 2 board marks and entrance test marks.
A discussion on social media quickly escalated into comparison with the prevalent system in developed countries; the most referenced being US and UK.
First let us take a look at the British system. When it comes to UK, Oxbridge serves as the gold standard. Here they don't have any standardized testing mechanism but they have a host of subject specific entrance tests. Each of these tests are tuned to the requirements of the course that the aspiring students wishes to pursue at University level.
India still follows standard and practices set by the British as benchmark and education is no exception. And thus many educationists support the implementation of this system in India as well.
The US system is a bit different. US follows the standardized testing approach for admission almost at all levels. For undergraduate courses SAT (Scholastic Aptitude Test) and ACT (American College Testing) are the most popular and widely accepted test scores.
How do these two widely different systems stack up against each other. I would say they differ in the basic premise; while the UK system tries to gauge 'know-how', the US system tries to measure 'ability' of the aspirant. The UK system considers that students who have better knowledge on a subject are more suited to follow the course at University level. Thus a student who best knows school level English should pursue English in University.
The US system is a bit more open. Here they measure the general ability of the student and how prepared they are to take a university course. A student with good SAT score would have a lot more options open in front of him to choose from. Thus with a good SAT score you can pursue English, Physics or any other subject in University. Compare this against the UK system, if a student wants to choose between English and Physics he/she would have to sit in two different examinations with widely different coverage area in terms of content. I suppose the US system presents students with lot more opportunities to choose from along with making it even more less taxing for them
Another important factor which should stand in favor of a standardized test like SAT is the ability to reduce bias. As we discussed in the beginning not all examinations favor every student equally. Thus more the number of tests and examination you have in the system, the more biased it is and hence more difficult it becomes to perfectly judge which candidate is most suitable.
However, it is not an easy task to reduce bias in examination to not unduly favor a certain section of students. Generally, the people who set the questions for the entrance tests believe they are doing the best job by being morally righteous. But even as they don't notice or understand they could be helping some section of students more than others. Those trends can only come in picture when detailed statistical analysis of the data from the examinations is done.
Let's learn how the same is done in case of SAT. In case SAT historically it has been observed that students belonging to Black/African-American ethnicity scored less than the White students. Initially it was thought that test questions placed the Black students at disadvantage compared to White students.
Educational Testing Services (ETS), company which owns and conducts SAT, was sued and the examiners, people who set the questions, were held at fault. However extensive data collection and their statistical analysis pointed to a basic fallacy in the underlying conclusion.
It was found incorrect to compare the average score of Black and White students and that is because both the group would have students of different abilities. And comparing a White student with higher abilities with Black student of lesser abilities would definitely be not correct. It was thus decided that while comparing scoring equal ability students from either groups should be considered.
When students with like abilities were compared then also it was found that some questions were biased towards a section of students than others. Statisticians employed a technique called Differential Item Functioning (DIF) to weed out the 'items' which exhibited biased behavior. Some such instances are given below as examples. It is hard to explain why some questions are unfavorable to certain group of students but data definitely points in the direction.
The general objective of SAT examiners is that at an overall level no question should exceed 40% difficulty level i.e. at least 40% of the students should have got that question correct. And no group of students should be more than 15% disadvantaged than another group i.e. if 60% of white students got an answer correct then at least 45% of Black students must have got that correct too.
SAT employs a huge pool of teachers and experts who work all through the year to contribute new questions to the bank. And then it has got a team of mathematicians and statisticians enabled by powerful computers and program to analyze those questions to remove out any bias before they are actually fed into the test. In the SAT examination as well students are made to take an experimental section, the answers to those questions don't impact the final score but are used for DIF analysis.
Thus the entire process of conducting a test and making it to suit the needs of a wide array students taking it is not a task to be achieved easily achieved. The formulation of the question, the options given and even the way the options are arranged can play its own role to make it biased.
In India as we see even some of the colleges / universities devising their own testing mechanism and can appreciated where it is going wrong. It could just be that students having access to better resources and reference materials are qualifying and not exactly the students who are with better abilities.
A case in point is the success of coaching institutes in Kota, Rajasthan in terms of clearing IIT JEE. In 2017, 45 out of the top 100 students were from Kota based coaching centers; 5 out of top 10 were from Kota as well. It clearly indicates that we have devised a system where students who have the resources and means to join one of those institutes deserve more to be an IIT-ian. And on the other side board examinations, which are taken by a wider section of students and wherein we see even students from remote and backward areas performing good, have totally lost their significance.
Students who rank better in competitive exams but score poorly in board examinations are cited as examples on the uselessness of the board results. But ever have we tried to dig deep why a student who is a star performer in board examinations had a poor show in the competitive; is it because of lack of ability or lack of resources.
In a country like ours where the diversity is more the chances are really high that we are placing an entry barrier in front of institutes of higher education which are easier for some than others not because they are better able to do so but because they have better tools to achieve the goal. Unfortunately, not enough data is recorded and analyzed about examinations in India; we are only concerned about the final percentage or rank. And till that time we wouldn't know whom we are robing; the students of opportunity or the country of talent.
Example 1
Following is one variant of the question for which girls were found to be 11% disadvantaged than boys.
In the past the general had been _____ for his emphasis on defensive strategies, but he was ______ when doctrines emphasizing aggression were discredited.
a> criticized, discharged
b> parodied, ostracized
c> supported, disappointed
d> spurned, vindicated (*)
e> praised, disregarded
Below is an altered version of the above question for which the group difference fell from 11 to 5%.
Heretofore ____ for her emphasis on conservation, the economist was ____ when doctrines emphasizing consumption were discredited.
a> criticized, discharged
b> parodied, ostracized
c> supported, disappointed
d> spurned, vindicated (*)
e> praised, disregarded
Example 2
For the below question it was found that overall difficulty level was 47% and black students were at an advantage than white students by 21%.
Plait: Hair
a> knead:bread
b> weave:yarn (*)
c> cut:cloth
d> fold:paper
e> frame:picture
Following is an altered version of the above question. The group difference is not visible in this case and the overall difficulty level rose to 80%.
Braid: Hair
a> knead:bread
b> weave:yarn (*)
c> cut:cloth
d> fold:paper
e> frame:picture
Example: 3
This question is observed to have an overall correctness of 80% and no noticeable difference between groups.
Dye:Fabric
a> thinner:stain
b> oil:skin
c> paint:wood(*)
d> fuel:engine
e> ink:pen
However, a slightly altered version of the above question introduced a DIF of 11-15% in favor of the white students.
Dye:Fabric
a> thinner:stain
b> oil:skin
c> stain:wood(*)
d> fuel:engine
e> ink:pen
References:
Thus it is important that tests organized for admission to these institutes are unbiased to the maximum extent possible; any bias could provide an undue advantage to one group of students over the other. In India there are a number of entrance examinations conducted some at an all India level and some at State levels. And for each of them we have quite often heard how one section of students is at disadvantage over the other. For example, it is widely known that students pursuing plus 2 in CBSE board are more at advantage (because of the syllabus content and coverage) than the students pursuing a State board for plus 2 study. And hence to counter this the State level competitive examinations are tuned to the State board syllabus.
Now this is the case for entrance examinations for engineering and medical colleges across the country. It is even more complicated for admission to Science and Humanities courses in colleges.
Because of the wide variety of Secondary and Higher Secondary education boards across the country, there is a wide variance in the marking pattern and the marks obtained by the students. Some of the colleges ignore such a ground reality and consider the marks obtained by the students in Plus 2 as-is on their face value. While others have their own algorithm to normalize the marks obtained by students in different board examinations. And then there are some other colleges/universities who further tax the students by asking them to their specific entrance examinations.
A case in point is that of Jadavpur University (JU), Kolkata which few weeks back went through a period of turmoil on this issue of admission test. The admission test was initially scrapped but then under pressure from students the tests were restored. However now another set of controversy has struck the University with a section of aspiring students having raised the issue of marks they got in the university conducted admission tests; they scored more than 90% in board examinations but got single digit marks in the JU entrance test.
So the question now is which examination to rely upon in order to judge the ability of the aspirants and their readiness for a University course. Some say it's the board examination and some others say it is the college / course specific entrance tests. And so the middle path is mostly followed; considering both Plus 2 board marks and entrance test marks.
A discussion on social media quickly escalated into comparison with the prevalent system in developed countries; the most referenced being US and UK.
First let us take a look at the British system. When it comes to UK, Oxbridge serves as the gold standard. Here they don't have any standardized testing mechanism but they have a host of subject specific entrance tests. Each of these tests are tuned to the requirements of the course that the aspiring students wishes to pursue at University level.
India still follows standard and practices set by the British as benchmark and education is no exception. And thus many educationists support the implementation of this system in India as well.
The US system is a bit different. US follows the standardized testing approach for admission almost at all levels. For undergraduate courses SAT (Scholastic Aptitude Test) and ACT (American College Testing) are the most popular and widely accepted test scores.
How do these two widely different systems stack up against each other. I would say they differ in the basic premise; while the UK system tries to gauge 'know-how', the US system tries to measure 'ability' of the aspirant. The UK system considers that students who have better knowledge on a subject are more suited to follow the course at University level. Thus a student who best knows school level English should pursue English in University.
The US system is a bit more open. Here they measure the general ability of the student and how prepared they are to take a university course. A student with good SAT score would have a lot more options open in front of him to choose from. Thus with a good SAT score you can pursue English, Physics or any other subject in University. Compare this against the UK system, if a student wants to choose between English and Physics he/she would have to sit in two different examinations with widely different coverage area in terms of content. I suppose the US system presents students with lot more opportunities to choose from along with making it even more less taxing for them
Another important factor which should stand in favor of a standardized test like SAT is the ability to reduce bias. As we discussed in the beginning not all examinations favor every student equally. Thus more the number of tests and examination you have in the system, the more biased it is and hence more difficult it becomes to perfectly judge which candidate is most suitable.
However, it is not an easy task to reduce bias in examination to not unduly favor a certain section of students. Generally, the people who set the questions for the entrance tests believe they are doing the best job by being morally righteous. But even as they don't notice or understand they could be helping some section of students more than others. Those trends can only come in picture when detailed statistical analysis of the data from the examinations is done.
Let's learn how the same is done in case of SAT. In case SAT historically it has been observed that students belonging to Black/African-American ethnicity scored less than the White students. Initially it was thought that test questions placed the Black students at disadvantage compared to White students.
Educational Testing Services (ETS), company which owns and conducts SAT, was sued and the examiners, people who set the questions, were held at fault. However extensive data collection and their statistical analysis pointed to a basic fallacy in the underlying conclusion.
It was found incorrect to compare the average score of Black and White students and that is because both the group would have students of different abilities. And comparing a White student with higher abilities with Black student of lesser abilities would definitely be not correct. It was thus decided that while comparing scoring equal ability students from either groups should be considered.
When students with like abilities were compared then also it was found that some questions were biased towards a section of students than others. Statisticians employed a technique called Differential Item Functioning (DIF) to weed out the 'items' which exhibited biased behavior. Some such instances are given below as examples. It is hard to explain why some questions are unfavorable to certain group of students but data definitely points in the direction.
The general objective of SAT examiners is that at an overall level no question should exceed 40% difficulty level i.e. at least 40% of the students should have got that question correct. And no group of students should be more than 15% disadvantaged than another group i.e. if 60% of white students got an answer correct then at least 45% of Black students must have got that correct too.
SAT employs a huge pool of teachers and experts who work all through the year to contribute new questions to the bank. And then it has got a team of mathematicians and statisticians enabled by powerful computers and program to analyze those questions to remove out any bias before they are actually fed into the test. In the SAT examination as well students are made to take an experimental section, the answers to those questions don't impact the final score but are used for DIF analysis.
Thus the entire process of conducting a test and making it to suit the needs of a wide array students taking it is not a task to be achieved easily achieved. The formulation of the question, the options given and even the way the options are arranged can play its own role to make it biased.
In India as we see even some of the colleges / universities devising their own testing mechanism and can appreciated where it is going wrong. It could just be that students having access to better resources and reference materials are qualifying and not exactly the students who are with better abilities.
A case in point is the success of coaching institutes in Kota, Rajasthan in terms of clearing IIT JEE. In 2017, 45 out of the top 100 students were from Kota based coaching centers; 5 out of top 10 were from Kota as well. It clearly indicates that we have devised a system where students who have the resources and means to join one of those institutes deserve more to be an IIT-ian. And on the other side board examinations, which are taken by a wider section of students and wherein we see even students from remote and backward areas performing good, have totally lost their significance.
Students who rank better in competitive exams but score poorly in board examinations are cited as examples on the uselessness of the board results. But ever have we tried to dig deep why a student who is a star performer in board examinations had a poor show in the competitive; is it because of lack of ability or lack of resources.
In a country like ours where the diversity is more the chances are really high that we are placing an entry barrier in front of institutes of higher education which are easier for some than others not because they are better able to do so but because they have better tools to achieve the goal. Unfortunately, not enough data is recorded and analyzed about examinations in India; we are only concerned about the final percentage or rank. And till that time we wouldn't know whom we are robing; the students of opportunity or the country of talent.
Example 1
Following is one variant of the question for which girls were found to be 11% disadvantaged than boys.
In the past the general had been _____ for his emphasis on defensive strategies, but he was ______ when doctrines emphasizing aggression were discredited.
a> criticized, discharged
b> parodied, ostracized
c> supported, disappointed
d> spurned, vindicated (*)
e> praised, disregarded
Below is an altered version of the above question for which the group difference fell from 11 to 5%.
Heretofore ____ for her emphasis on conservation, the economist was ____ when doctrines emphasizing consumption were discredited.
a> criticized, discharged
b> parodied, ostracized
c> supported, disappointed
d> spurned, vindicated (*)
e> praised, disregarded
Example 2
For the below question it was found that overall difficulty level was 47% and black students were at an advantage than white students by 21%.
Plait: Hair
a> knead:bread
b> weave:yarn (*)
c> cut:cloth
d> fold:paper
e> frame:picture
Following is an altered version of the above question. The group difference is not visible in this case and the overall difficulty level rose to 80%.
Braid: Hair
a> knead:bread
b> weave:yarn (*)
c> cut:cloth
d> fold:paper
e> frame:picture
Example: 3
This question is observed to have an overall correctness of 80% and no noticeable difference between groups.
Dye:Fabric
a> thinner:stain
b> oil:skin
c> paint:wood(*)
d> fuel:engine
e> ink:pen
However, a slightly altered version of the above question introduced a DIF of 11-15% in favor of the white students.
Dye:Fabric
a> thinner:stain
b> oil:skin
c> stain:wood(*)
d> fuel:engine
e> ink:pen
References:
- https://www.ucas.com/undergraduate/applying-university/admissions-tests
- https://blog.ivywise.com/blog-0/us-vs.-uk-admissions-what-parents-and-students-need-to-know
- https://blog.prepscholar.com/what-is-the-average-sat-score
- https://blog.prepscholar.com/what-is-the-sat-experimental-section
- Wikipedia
- Numbers Rule Your World – Kaiser Fung (example are from this book)
- https://timesofindia.indiatimes.com/india/iit-jee-advanced-results-decoded/articleshow/59109911.cms
No comments:
Post a Comment