Grade the Bot!
Dave Burgess (Twitter: @dbc_inc) and his team have a great idea for Educators to leverage ChatGPT in the classroom. Grade the bot!
The Good
ChatGPT has successfully answered many of my semi-technical questions. YouTube has many videos of particularly challenging questions, and the ability to drill-down into ChatGPT answers. The grammar, syntax and readability of the responses have been very satisfactory (a little editing is to be expected). If this project gains access to larger, more current data, the results will be amazing.
The Bad
The exiting (limited) dataset can be exposed by asking questions about current topics. Some topics associated with Social Security stumped ChatGPT. Responses to a tricky question "I'm 10 years older than my wife. What are suggestions for maximizing my social security check while protecting her income?" came back with a very generic answer and a suggestion to "check with a social security expert". When the question was reworded, the answers were very much the same. Similar "weak" responses for COVID related questions were noted.
The Ugly
The inability to understand Government programs like social security - at least for this version of ChatGPT, is an interesting blind spot. While many have pointed out that the data sources are unknown, the current narrative is that ChatGPT will skew liberal to reflect existing online content.
The addition of moderators will introduce a new collection of bias. Moderators will have the challenge of being fair, without polarizing the question response sets. One person's conspiracy theory is another person's truth (and vice-versa).