On Using ChatGPT in the Biblical Studies Classroom

The alternative title for this post is: “How ChatGPT made me look like an idiot.”

I initially wrote this post to show ChatGPT could be used in the Biblical Studies classroom. But after experimenting with it, and thanks to the critical comments of Tyler Jarvis (!), I realized that there are still some major problems, specifically with referencing primary texts and accessing secondary literature.

I’ll admit at the start that I was initially completely duped. Anyone who knows how I do research knows I’m a “turtles all the way down” kind of person, which means I always double and triple check the sources of what scholars say they’re saying. I don’t trust that anyone has necessarily said or quoted the right thing. With ChatGPT, I was lulled into a false sense of security, which I will talk about below.

In my exegesis classes, we have weekly debates centred around a particular topic. This week in my Romans class we debated the following topic: “Paul is speaking only to Gentiles in Romans 2. Discuss.” Those of you well-versed in the arcana of Pauline studies know that this is an ongoing conversation in the field passing through the important works of scholars like Stanley Stowers and the Paul within Judaism scene. Undergraduate students, on the other hand, aren’t necessarily privy to these discourses and so penetrating them can be very difficult. I tried using ChatGPT to explore key texts, access primary texts, and access secondary scholarship.

I asked the AI: "Is Paul speaking only to gentiles in Romans 2?" 

It responded: "In Romans 2, Paul addresses both Gentile and Jewish audiences. He begins by addressing the Gentiles in verse 1, warning them not to pass judgment on others while they themselves commit similar sins. He then turns to the Jews in verse 17, reminding them that possessing knowledge of the law does not make them righteous and that they too will be judged according to their actions. Throughout the chapter, Paul emphasizes that righteousness comes from obeying God's law, not from being a member of a particular ethnic group or from having knowledge of the law."

After putting a variation of the debate statement into ChatGPT, it gives an answer providing two pieces of textual support. It thinks that Paul is addressing both Jewish and non-Jewish audiences based on:

  • Romans 2:1 – Paul is addressing gentiles
  • Romans 2:17 – Paul is addressing Jews

Now so far these are just assertions, and I’m teaching my students to move beyond mere assertions, to justify them with textual evidence and supporting arguments. ChatGPT provides verse support in Romans 2, but again only asserts an interpretation of these verses. If I was using this in the classroom, I’d encourage students to probe further in the commentaries to verify whether in fact Romans 2:1 and 2:17 refer to Gentiles and Jews respectively. When students consult commentaries on these passages they will then get some context for the ambiguity of 2:1 (the language of “O human who judges”) and how many interpreters think that Paul is referring to a Jewish person judging the gentile behaviour of Romans 1:18-32. Students will also get some context for Romans 2:17 and the way Paul seems to refer to a Jewish person but how the expression “if you call yourself” is strange (why would a Jewish person call themselves anything other than a Jew?). I recommend giving a supplementary reading for them to explore in addition the commentary material like the chapter “Paul’s Interlocutor in Romans” by Thorsteinsson, Thiessen, and Rodríguez.

With ChatGPT you can inquire further about Romans 2:17:

I asked the AI: "How do you know that Paul is referring to Jews in Romans 2:17?" It responded: "In Romans 2:17, Paul uses the Greek word Ἰουδαίων (Ioudaion) which is the plural form of Ioudaios (Jew). He also refers to those he is addressing in this verse as "you who boast in the law" and "you who boast in God", which are typical expressions used to refer to Jews, as they were the ones who possessed the law and had a special relationship with God as his chosen people. Additionally, Paul also refers to the "circumcision" which is a term that is used to refer to the physical sign of the covenant between God and the Israelites, this is another indication that he is addressing Jewish audience."

I pressed ChatGPT further. Its response, though seemingly well-founded, is actually quite a weak argument. Just because Paul uses the terms “Ioudaios” and “circumcision” doesn’t mean he’s necessarily addressing someone Jewish. He talks about both Jews and circumcision in Galatians and is clearly speaking to a gentile community. “You who boast in the law” in Rom 2:23 is a stronger piece of evidence, but as students will know after reading Thorsteinsson, Thiessen, and Rodríguez, judaising gentiles or proselytes might also boast in the Jewish law. Based on this you could inquiry further with ChatGPT:

I asked the AI: "Could he not also be referring to gentile proselytes who have circumcised?" It responded: "Yes, it is possible that Paul could also be referring to Gentile converts to Judaism who have undergone circumcision as a sign of their adherence to the Jewish faith. In the 1st century AD, there were many Gentile converts to Judaism who were called "proselytes" or "God-fearers" and they would also follow the laws of the Jews, including the circumcision. It is important to note that Paul's audience in Rome would have been a mixed one, with both Jewish and Gentile converts, and he would have been aware of the presence of these God-fearers in the community. So, it's plausible that he could be addressing both groups in this passage."

ChatGPT’s response touches on three important areas of ancient Jewish studies that undergrads might not be aware of, first the idea of “conversion” itself; the nature of gentile “conversion” to Judaism (a good resource for students would be Cohen’s article here), and whether or not ancient Judaism was a “missionary” “religion” at all (key works here involve the work of Mike Bird, John Dickson, and Scot McKnight, but my old teacher Martin Goodman clinches that discussion IMHO).

Again, ChatGPT is making assertions. For students to succeed in my classes they need to provide support. Students should become familiar with “proselytes” to Judaism and the sources that talk about them. You might encourage them to inquire the AI further:

I asked the AI: "You say there were “many converts”. What ancient sources and evidence do we have for this?" and it responded: "There are several ancient sources that provide evidence for the presence of Gentile converts to Judaism in the 1st century AD. The most important source is the New Testament itself, which contains several references to Gentile converts to Judaism, particularly in the book of Acts, where it describes the early spread of Christianity among both Jews and Gentiles. For example, in Acts 10:1-11:18, it tells the story of the conversion of the Roman centurion Cornelius and his household, who become the first Gentile converts to Christianity, and in Acts 13:16-41, it describes Paul's mission to the Gentiles in Antioch.

Another source that provides evidence for the presence of Gentile converts to Judaism is the Jewish historian Flavius Josephus. In his work "The Antiquities of the Jews," he writes about the existence of Gentile converts to Judaism who were called "God-fearers" and were attracted to the Jewish religion but not fully converted. He also mentions that there were many Gentiles who circumcised themselves and observed the Sabbath and other Jewish customs as a way of showing their devotion to the Jewish God.

Additionally, there are also some epigraphic and archaeological evidence that confirms the presence of God-fearers in the ancient world. For example, inscriptions and synagogue art have been found that depict Gentile donors and benefactors, who have been interpreted as God-fearers.

All of these sources indicate that there were many Gentile converts to Judaism in the 1st century AD, and it's likely that Paul would have been aware of their presence in the Roman community.
It is important to note that the number of converts and the degree of their adherence to Jewish customs and laws varied and are difficult to estimate. The idea of a "God-fearer" could also mean different things in different communities. However, the sources I mentioned above give us a glimpse into the cultural and religious diversity of the ancient world and show that the presence of Gentile converts to Judaism was a significant phenomenon in the 1st century AD."

This is much better, at least there are some sources posted. But it doesn’t give specific references or translations of Josephus or inscriptions. We can ask it to be more specific.

I asked the AI: "Can you cite specific sources and translations for sources of gentiles converting to Judaism in the ancient world?" and it responded: "There are several ancient sources that mention gentiles converting to Judaism in the ancient world. Here are a few examples:

Josephus, Jewish Antiquities, 18.14-16, 20.2.2: In these passages, Josephus mentions the conversion of a number of gentiles to Judaism, including the King of Adiabene and his family, who were converted by a Jewish sage named Ananias.
Philo, Special Laws, 2.26-31: In this passage, Philo describes the process of conversion to Judaism, including the requirement of circumcision for men and the adoption of Jewish customs and laws.
The book of Acts: In several passages in the book of Acts, it is mentioned that gentiles converts to Judaism, such as Cornelius (Acts 10) and the Ethiopian eunuch (Acts 8:26-40)
The Talmud: There are also many references to gentiles converting to Judaism in the Talmud, which is a collection of Jewish laws and teachings written in the centuries after the time of Jesus.
These sources are ancient texts and are not easily accessible by most people. Some translations of these sources can be found in books like "The Works of Philo" translated by C.D. Yonge, "The Jewish War" translated by G.A. Williamson, "Acts of the Apostles" translated by F.F. Bruce, "The Babylonian Talmud" translated by I. Epstein

It's worth noting that it's difficult to know how many people converted and how common it was in the ancient world, as it's likely that many converts were not recorded in historical texts."

That appears much better (see below), but of course they need to be double checked in commentaries. The “conversion” of the Ethiopian eunuch for example in Acts 8 I think may be debatable. But again, these are only textual references. If students don’t know how to work with Josephus’s and Philo’s corpora which are vast and which have different versification depending on edition, it can be tricky to verify these sources directly. Again, ChatGPT is asserting evidence without explanation. That’s where you’ll need to direct students to Safaria or Loeb or other translations of the works.

Finally, you might ask ChatGPT for some secondary scholarship on conversion in ancient Judaism:

I asked the AI "Can you recommend some good peer-reviewed scholarship on conversion and ancient Judaism?" and it responded "There are many peer-reviewed scholarly works on conversion and ancient Judaism that are worth reading. Here are a few examples:

"Conversion to Judaism in the First Century CE" by Paula Fredriksen. This article, published in the Journal of Early Christian Studies, discusses the nature of conversion to Judaism in the first century CE, focusing on the issue of circumcision as a marker of conversion.
"The God-Fearers: A Study of the Relationship between Jews and Gentiles in the Early Synagogue" by David Flusser. This book, published by the Hebrew University Magnes Press, examines the phenomenon of the God-fearers, a group of Gentiles who were attracted to Judaism but not fully converted.
"The Parting of the Ways: The Roman Church and the Jewish Community after 70 CE" by James D. G. Dunn. This book, published by Westminster John Knox Press, examines the emergence of Christianity as a separate religion from Judaism, focusing on the role of Gentile converts in the formation of the early Christian community.
"Paul, Judaism, and the Gentiles: A Sociological Approach" by J. Andrew Overman. This book, published by Cambridge University Press, uses sociological analysis to examine Paul's relationship with Judaism and his mission to the Gentiles.
"The Jews and the Gentiles in the Diaspora: An Analysis of the Halakhic Sources" by Yaakov Elman. This article, published in the Journal of Jewish Studies, examines the Halakhic sources of the period and their discussion of the status of the God-fearers and other gentiles in the Jewish community.
These are just a few examples of the many scholarly works that are available on this topic. I recommend checking the bibliographies of these works for further reading."

I did go further by asking the question more specifically.

I asked the AI: "Can you provide some good peer-reviewed scholarship on gentile conversion in ancient Judaism?" and it responded: "Certainly, here are a few examples of peer-reviewed scholarship on gentile conversion in ancient Judaism:

"Proselytism in the Second Temple Period" by George J. Brooke, Journal for the Study of Judaism, Vol. 19, No. 3 (1988), pp. 211-234.
"God-Fearers and the Jewish Community in the Graeco-Roman Diaspora" by Robert D. Stock, Journal of Jewish Studies, Vol. 42, No. 2 (Autumn, 1991), pp. 213-232
"The Significance of Proselytes and Proselytism in Second Temple Judaism" by David A. deSilva, Journal for the Study of Judaism in the Persian, Hellenistic, and Roman Period, Vol. 29, No. 3 (1998), pp. 337-367
"Gentile Conversion to Judaism in the Roman Period" by David B. Levenson, The Jewish Quarterly Review, Vol. 99, No. 4 (2009), pp. 513-558
"Proselytes and God-Fearers in the Synagogue: On the Probable Nature of Ancient Jewish Proselytism" by Magnus Zetterholm, Journal for the Study of Judaism in the Persian, Hellenistic, and Roman Period, Vol. 37, No. 3 (2006), pp. 320-342
You can find these articles in academic libraries, university libraries, and online research databases such as JSTOR and Project MUSE."

As a final inquiry leading us back to Romans 2, students can ask about scholarship directly related to Romans 2 and gentile proselytes or judaising gentiles (I used the former because I wasn’t sure if the AI would be able to understand what I mean by that):

I asked the AI: "Can you recommend some scholarship that views Romans 2 as speaking only to gentile proselytes" and it responded: "There is a significant amount of scholarship that interprets Romans 2 as addressing gentile proselytes. Some notable examples include:

"Paul's Letter to the Romans" by Joseph A. Fitzmyer, which argues that Romans 2 is primarily addressed to gentile converts to Judaism.
"The Epistle to the Romans" by James D.G. Dunn, which also suggests that Romans 2 is primarily addressing gentile converts.
"Paul's Gentile-Jews: Neither Jew nor Gentile" by Mark Nanos, which argues that the audience of Romans 2 is made up of gentile converts who are in the process of becoming Jewish.
It is important to note that interpretation of Romans 2 is a subject of ongoing scholarly debate, and different scholars have different perspectives on the intended audience of the letter."

These appear to be good results by reputable scholars. But after a closer look, none of these works actually exists. ChatGPT actually generates imagined resources that look reputable but cannot actually be found.

Honestly, when I first looked at this I only took a cursory look and was misled. I saw especially James Dunn’s book “The Parting of the Ways” which is actually a book, but published by a different press and with a completely different subtitle. Fitzmyer does have a commentary on Romans as does James Dunn. “Paul’s Gentile-Jews” is actually a book by Joshua Garroway, not Mark Nanos. And ChatCPT is inconsistent, as it does mention actual translations of Philo and Josephus by Yonge and Williamson in an earlier question above. Based on these sources, I presumed (wrongly) that all the sources were good. My spider senses did kick off, however, seeing initially the alleged article by Paula Fredriksen, which sounded like something she would write, but I was not familiar with. But again, I don’t have a perfect memory and an exhaustive knowledge of what every NT scholar has published.

What makes detecting this difficult at first, even for scholars in the field (i.e., me!), is the realistic way the references are generated. Reputable journals and publishers are given, years, issue editions, even page numbers. In my haste this morning, I didn’t double check the references, despite hesitating about some of them (I repent!).

This made me turn back to the primary text references that ChatGPT generated earlier in Josephus, Philo, and Acts. The Acts 8 and 10 references “kind of” check out, depending on if you few the Jesus movement as a Jewish movement. But looking up he passages given in Philo’s Special Laws rendered no such discussion on proselytism. ChatGPT does reference the right section in Josephus, however, for the discussion of the conversion of the king of Adiabene.

We’ll stop there. I initially thought that it would be a good tool to engage in primary and secondary scholarship with, but since finding out that the AI generates fake scholarship sources and primary source references, I now am hesitant to use it at all with students.


  1. This is brilliant, Isaac. I appreciate your calmly looking at this new technology as the tool it is, devising already a way to use it for and among your students, and then setting out in this lucid way a pattern for its use. Excellent pedagogy from which I am gratefully learning!


    1. Angelo Giovas says:

      I also appreciate your work on this and it does give pause for concern.
      If I may I would in a gentle way also say that there are some (numerous) grammatical fluffs in your article. :^)
      Merely to say that if that was an email that came through my in box it would be considered for the spam folder :^})>
      But I appreciate your work here.


Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s