On Using ChatGPT in the Biblical Studies Classroom

The alternative title for this post is: “How ChatGPT made me look like an idiot.”

I initially wrote this post to show ChatGPT could be used in the Biblical Studies classroom. But after experimenting with it, and thanks to the critical comments of Tyler Jarvis (!), I realized that there are still some major problems, specifically with referencing primary texts and accessing secondary literature.

I’ll admit at the start that I was initially completely duped. Anyone who knows how I do research knows I’m a “turtles all the way down” kind of person, which means I always double and triple check the sources of what scholars say they’re saying. I don’t trust that anyone has necessarily said or quoted the right thing. With ChatGPT, I was lulled into a false sense of security, which I will talk about below.

In my exegesis classes, we have weekly debates centred around a particular topic. This week in my Romans class we debated the following topic: “Paul is speaking only to Gentiles in Romans 2. Discuss.” Those of you well-versed in the arcana of Pauline studies know that this is an ongoing conversation in the field passing through the important works of scholars like Stanley Stowers and the Paul within Judaism scene. Undergraduate students, on the other hand, aren’t necessarily privy to these discourses and so penetrating them can be very difficult. I tried using ChatGPT to explore key texts, access primary texts, and access secondary scholarship.

After putting a variation of the debate statement into ChatGPT, it gives an answer providing two pieces of textual support. It thinks that Paul is addressing both Jewish and non-Jewish audiences based on:

• Romans 2:1 – Paul is addressing gentiles
• Romans 2:17 – Paul is addressing Jews

Now so far these are just assertions, and I’m teaching my students to move beyond mere assertions, to justify them with textual evidence and supporting arguments. ChatGPT provides verse support in Romans 2, but again only asserts an interpretation of these verses. If I was using this in the classroom, I’d encourage students to probe further in the commentaries to verify whether in fact Romans 2:1 and 2:17 refer to Gentiles and Jews respectively. When students consult commentaries on these passages they will then get some context for the ambiguity of 2:1 (the language of “O human who judges”) and how many interpreters think that Paul is referring to a Jewish person judging the gentile behaviour of Romans 1:18-32. Students will also get some context for Romans 2:17 and the way Paul seems to refer to a Jewish person but how the expression “if you call yourself” is strange (why would a Jewish person call themselves anything other than a Jew?). I recommend giving a supplementary reading for them to explore in addition the commentary material like the chapter “Paul’s Interlocutor in Romans” by Thorsteinsson, Thiessen, and Rodríguez.

With ChatGPT you can inquire further about Romans 2:17:

I pressed ChatGPT further. Its response, though seemingly well-founded, is actually quite a weak argument. Just because Paul uses the terms “Ioudaios” and “circumcision” doesn’t mean he’s necessarily addressing someone Jewish. He talks about both Jews and circumcision in Galatians and is clearly speaking to a gentile community. “You who boast in the law” in Rom 2:23 is a stronger piece of evidence, but as students will know after reading Thorsteinsson, Thiessen, and Rodríguez, judaising gentiles or proselytes might also boast in the Jewish law. Based on this you could inquiry further with ChatGPT:

ChatGPT’s response touches on three important areas of ancient Jewish studies that undergrads might not be aware of, first the idea of “conversion” itself; the nature of gentile “conversion” to Judaism (a good resource for students would be Cohen’s article here), and whether or not ancient Judaism was a “missionary” “religion” at all (key works here involve the work of Mike Bird, John Dickson, and Scot McKnight, but my old teacher Martin Goodman clinches that discussion IMHO).

Again, ChatGPT is making assertions. For students to succeed in my classes they need to provide support. Students should become familiar with “proselytes” to Judaism and the sources that talk about them. You might encourage them to inquire the AI further:

This is much better, at least there are some sources posted. But it doesn’t give specific references or translations of Josephus or inscriptions. We can ask it to be more specific.

That appears much better (see below), but of course they need to be double checked in commentaries. The “conversion” of the Ethiopian eunuch for example in Acts 8 I think may be debatable. But again, these are only textual references. If students don’t know how to work with Josephus’s and Philo’s corpora which are vast and which have different versification depending on edition, it can be tricky to verify these sources directly. Again, ChatGPT is asserting evidence without explanation. That’s where you’ll need to direct students to Safaria or Loeb or other translations of the works.

Finally, you might ask ChatGPT for some secondary scholarship on conversion in ancient Judaism:

I did go further by asking the question more specifically.

As a final inquiry leading us back to Romans 2, students can ask about scholarship directly related to Romans 2 and gentile proselytes or judaising gentiles (I used the former because I wasn’t sure if the AI would be able to understand what I mean by that):

These appear to be good results by reputable scholars. But after a closer look, none of these works actually exists. ChatGPT actually generates imagined resources that look reputable but cannot actually be found.

Honestly, when I first looked at this I only took a cursory look and was misled. I saw especially James Dunn’s book “The Parting of the Ways” which is actually a book, but published by a different press and with a completely different subtitle. Fitzmyer does have a commentary on Romans as does James Dunn. “Paul’s Gentile-Jews” is actually a book by Joshua Garroway, not Mark Nanos. And ChatCPT is inconsistent, as it does mention actual translations of Philo and Josephus by Yonge and Williamson in an earlier question above. Based on these sources, I presumed (wrongly) that all the sources were good. My spider senses did kick off, however, seeing initially the alleged article by Paula Fredriksen, which sounded like something she would write, but I was not familiar with. But again, I don’t have a perfect memory and an exhaustive knowledge of what every NT scholar has published.

What makes detecting this difficult at first, even for scholars in the field (i.e., me!), is the realistic way the references are generated. Reputable journals and publishers are given, years, issue editions, even page numbers. In my haste this morning, I didn’t double check the references, despite hesitating about some of them (I repent!).

This made me turn back to the primary text references that ChatGPT generated earlier in Josephus, Philo, and Acts. The Acts 8 and 10 references “kind of” check out, depending on if you few the Jesus movement as a Jewish movement. But looking up he passages given in Philo’s Special Laws rendered no such discussion on proselytism. ChatGPT does reference the right section in Josephus, however, for the discussion of the conversion of the king of Adiabene.

We’ll stop there. I initially thought that it would be a good tool to engage in primary and secondary scholarship with, but since finding out that the AI generates fake scholarship sources and primary source references, I now am hesitant to use it at all with students.