The alternative title for this post is: “How ChatGPT made me look like an idiot.”
I initially wrote this post to show ChatGPT could be used in the Biblical Studies classroom. But after experimenting with it, and thanks to the critical comments of Tyler Jarvis (!), I realized that there are still some major problems, specifically with referencing primary texts and accessing secondary literature.
I’ll admit at the start that I was initially completely duped. Anyone who knows how I do research knows I’m a “turtles all the way down” kind of person, which means I always double and triple check the sources of what scholars say they’re saying. I don’t trust that anyone has necessarily said or quoted the right thing. With ChatGPT, I was lulled into a false sense of security, which I will talk about below.
In my exegesis classes, we have weekly debates centred around a particular topic. This week in my Romans class we debated the following topic: “Paul is speaking only to Gentiles in Romans 2. Discuss.” Those of you well-versed in the arcana of Pauline studies know that this is an ongoing conversation in the field passing through the important works of scholars like Stanley Stowers and the Paul within Judaism scene. Undergraduate students, on the other hand, aren’t necessarily privy to these discourses and so penetrating them can be very difficult. I tried using ChatGPT to explore key texts, access primary texts, and access secondary scholarship.
After putting a variation of the debate statement into ChatGPT, it gives an answer providing two pieces of textual support. It thinks that Paul is addressing both Jewish and non-Jewish audiences based on:
Romans 2:1 – Paul is addressing gentiles
Romans 2:17 – Paul is addressing Jews
Now so far these are just assertions, and I’m teaching my students to move beyond mere assertions, to justify them with textual evidence and supporting arguments. ChatGPT provides verse support in Romans 2, but again only asserts an interpretation of these verses. If I was using this in the classroom, I’d encourage students to probe further in the commentaries to verify whether in fact Romans 2:1 and 2:17 refer to Gentiles and Jews respectively. When students consult commentaries on these passages they will then get some context for the ambiguity of 2:1 (the language of “O human who judges”) and how many interpreters think that Paul is referring to a Jewish person judging the gentile behaviour of Romans 1:18-32. Students will also get some context for Romans 2:17 and the way Paul seems to refer to a Jewish person but how the expression “if you call yourself” is strange (why would a Jewish person call themselves anything other than a Jew?). I recommend giving a supplementary reading for them to explore in addition the commentary material like the chapter “Paul’s Interlocutor in Romans” by Thorsteinsson, Thiessen, and Rodríguez.
With ChatGPT you can inquire further about Romans 2:17:
I pressed ChatGPT further. Its response, though seemingly well-founded, is actually quite a weak argument. Just because Paul uses the terms “Ioudaios” and “circumcision” doesn’t mean he’s necessarily addressing someone Jewish. He talks about both Jews and circumcision in Galatians and is clearly speaking to a gentile community. “You who boast in the law” in Rom 2:23 is a stronger piece of evidence, but as students will know after reading Thorsteinsson, Thiessen, and Rodríguez, judaising gentiles or proselytes might also boast in the Jewish law. Based on this you could inquiry further with ChatGPT:
ChatGPT’s response touches on three important areas of ancient Jewish studies that undergrads might not be aware of, first the idea of “conversion” itself; the nature of gentile “conversion” to Judaism (a good resource for students would be Cohen’s article here), and whether or not ancient Judaism was a “missionary” “religion” at all (key works here involve the work of Mike Bird, John Dickson, and Scot McKnight, but my old teacher Martin Goodman clinches that discussion IMHO).
Again, ChatGPT is making assertions. For students to succeed in my classes they need to provide support. Students should become familiar with “proselytes” to Judaism and the sources that talk about them. You might encourage them to inquire the AI further:
This is much better, at least there are some sources posted. But it doesn’t give specific references or translations of Josephus or inscriptions. We can ask it to be more specific.
That appears much better (see below), but of course they need to be double checked in commentaries. The “conversion” of the Ethiopian eunuch for example in Acts 8 I think may be debatable. But again, these are only textual references. If students don’t know how to work with Josephus’s and Philo’s corpora which are vast and which have different versification depending on edition, it can be tricky to verify these sources directly. Again, ChatGPT is asserting evidence without explanation. That’s where you’ll need to direct students to Safaria or Loeb or other translations of the works.
Finally, you might ask ChatGPT for some secondary scholarship on conversion in ancient Judaism:
I did go further by asking the question more specifically.
As a final inquiry leading us back to Romans 2, students can ask about scholarship directly related to Romans 2 and gentile proselytes or judaising gentiles (I used the former because I wasn’t sure if the AI would be able to understand what I mean by that):
These appear to be good results by reputable scholars. But after a closer look, none of these works actually exists. ChatGPT actually generates imagined resources that look reputable but cannot actually be found.
Honestly, when I first looked at this I only took a cursory look and was misled. I saw especially James Dunn’s book “The Parting of the Ways” which is actually a book, but published by a different press and with a completely different subtitle. Fitzmyer does have a commentary on Romans as does James Dunn. “Paul’s Gentile-Jews” is actually a book by Joshua Garroway, not Mark Nanos. And ChatCPT is inconsistent, as it does mention actual translations of Philo and Josephus by Yonge and Williamson in an earlier question above. Based on these sources, I presumed (wrongly) that all the sources were good. My spider senses did kick off, however, seeing initially the alleged article by Paula Fredriksen, which sounded like something she would write, but I was not familiar with. But again, I don’t have a perfect memory and an exhaustive knowledge of what every NT scholar has published.
What makes detecting this difficult at first, even for scholars in the field (i.e., me!), is the realistic way the references are generated. Reputable journals and publishers are given, years, issue editions, even page numbers. In my haste this morning, I didn’t double check the references, despite hesitating about some of them (I repent!).
This made me turn back to the primary text references that ChatGPT generated earlier in Josephus, Philo, and Acts. The Acts 8 and 10 references “kind of” check out, depending on if you few the Jesus movement as a Jewish movement. But looking up he passages given in Philo’s Special Laws rendered no such discussion on proselytism. ChatGPT does reference the right section in Josephus, however, for the discussion of the conversion of the king of Adiabene.
We’ll stop there. I initially thought that it would be a good tool to engage in primary and secondary scholarship with, but since finding out that the AI generates fake scholarship sources and primary source references, I now am hesitant to use it at all with students.
Applying to study abroad, especially for graduate programs can be a daunting task. Although the internet has made application requirements, recommendation letter submissions, and deadlines more accessible than ever, there are still often unspoken (or vaguely-spoken) practices that can help give you an “advantage” in the application process for a graduate program in New Testament in the UK. I say “advantage,” because none of the advice that I give here will necessarily secure you a place or funding at a UK institution, only that such practices can help maximize the chances of your application. I’m also not trying to be exhaustive. If anything, I imagine this is helpful advice I don’t see sufficiently stressed elsewhere in the advice that I’ve read given online and in a few published books.
I’ve put a comments section at the bottom too for any of my colleagues to add more information or to nuance/disagree with me, so make sure to read what comes up there as well.
A Few Provisos
There are a few provisos that I feel obligated to state at the beginning.
The first is that I give this advice as someone who has gone through the process and learned from it myself, sometimes after the fact. I’ve also coached a few of my colleagues in their own graduate applications to the UK, many of whom have been successful.For this reason, the advice here is limited to UK PhD programs in New Testament and not to programs elsewhere, such as in Europe, Asia, Africa, Oceana, or North America. Even still, the advice given here may not be representative of all Theology & Religion departments in the UK. A repeated phrase you’ll see on this page is to make sure to read university websites thoroughly. There is a lot of information online and if you’re serious about programs then you need to do the hard yards of actually orienting yourself to the departments themselves. My suggestion is that you store program information in a readily available chart or organisational software like Notion. There are ready templates on there for job applications, for example, that you can use to organise graduate program deadlines, entry requirements, checklists, and documents.
The second proviso is that personally I do not think doing graduate study at the PhD level is necessarily a safe choice given the current job market if you intend to teach at a post-graduate level. You might raise the question about why I offer this advice here at all. My offering of advice (take it or leave it) is not an an endorsement of the practice. I give it in the case that you, Reader, decide to take the risk anyway. I cannot discourage you, but I can help with what little I might have. Make no mistake, it is very risky and costly and the vast majority of graduates who finish their PhD in NT will not get a full-time tenure track position. If you are ready for this, or are planning on shifting into full-time pastoral work for example that’s fine. But if you are seriously considering doing graduate work in New Testament then I would encourage you to from the beginning look at alternate-academic (alt-ac) careers and ways to apply your skills and accomplishments to more than just a single field/career (e.g. higher education).
The third proviso is that I do not recommend taking out loans to do PhD work (anywhere), but especially if you’re an international student in the UK. When I looked over tuition for graduate programs at the institutions I attended the range was between £20,000-30,000 ($24,000-35,000 USD) per year! Just for tuition. This especially applies if you have a family, if you have kids. Even if you have a job lined up after your PhD. You could have a full-time job lined up after your PhD in a city with exorbitant living costs and not pay off your student-loans or bank-loans for a decade. This is the advice I received from professors at top UK institutions and I have seen and heard stories of people who take out huge loans for a PhD program.
If these provisos haven’t discouraged you enough and you’re still reading, congratulations! Let’s go on to the advice.
You’re Probably Going to be Doing a Masters First
Wait, you’re thinking, I thought this was about PhD programs in New Testament. The reality is that unless you’ve studied biblical studies since your undergrad and have already done a masters with a significant dissertation component, you’ll probably be encouraged to do a Masters program before you do the PhD. Why is this the case?
Research PhDs in the UK are intense experiences that are mostly “hands-off.” You will have supervision meetings, yes, and you’ll have regular departmental check-ups in you first and final year, but for the most part you will be left to your own time management and research skills. If you don’t have a demonstrable history of doing independent research then you’re application is less likely to be successful and you’ll be directed toward a Masters program at the institution with a large dissertation component (between 15,000-30,000 words).
There are other reasons for why UK institutions encourage prospective international PhD students to do masters programs in addition to independent research. One is that their previous training has not sufficiently prepared them for independent research (in the eyes of the institution–I’m not making this judgment from my perspective). Language acquisition (both ancient and modern) relative to the project, knowledge of the current state-of-play of the field is also desirable. These are rarely ever gained from a single MA or Mth program (even less likely, if ever, from an MDiv alone). Even for those who have been lucky to study biblical studies from their undergraduate degree may need some extra time to expand their knowledge and gain more skills for the PhD.
I did my undergraduate degree majoring in biblical studies and did an 18-month MTh with a focus on New Testament studies. In my masters program I did at least (if I recall correctly), three independent guided studies (6-7,000 word papers on research questions myself) and a 20,000 word MTh dissertation. Even then, I still went on to do an MPhil in Theology at Oxford. It was one of the best things for my academic career. I had to really dive into ancient Jewish, Christian, and pagan sources; I had to learn how to write and argue effectively. The program was very intense — I was only marked after two years on three papers, my dissertation, and two 3 hour exams. But it gave me a very strong foundation and I had the opportunity to work with fantastic scholars, sit in cutting-edge seminars, and have access to resources from ancient manuscripts (like the Oxyrhynchus papyri) to the artefacts in the Ashmolean museum. It also allowed me to start building networks across Europe and extending into North America as well. Not every UK institution is the same, but all the Theology & Religion programs I know offer some version of this in different ways and combinations: Durham, Edinburgh, King’s College, Cambridge, Exeter, Nottingham, Manchester, Birmingham, Glasgow, the list goes on.
Keep in mind that a lot of Masters programs (MA, MTh, MSt, MLitt) are 9-12 months long. If you’re wanting to apply for PhD programs and funding from the Masters program, you’re pretty much going to be applying the first term of your program (Michaelmas) which can be stressful. A lot will depend on your research proposal and how it fits with the institution (as I note below). One of the benefits of the 2-year MPhil at Oxford was that it gave me time to find my feet and craft a really strong research proposal that would be competitive for funding.
The Research Proposal
For PhD programs, the research proposal is really an important part of your application. You’ve got to have strong writing examples, good recommendation letters (preferably even by scholars who might be on the radar of the department you’re applying to), and good grades.
Research proposals for NT PhD programs in the UK are about 1,000-1,500 words. That is really not a lot of space to convey your project, how it fits with the institution, and hopefully why it should be funded! So, you need to make sure that you’ve really worked hard on that proposal to make it as clear as possible.
Research Question. The first thing I’d recommend with the research proposal is that you should be consulting with your potential supervisor at the institution to which you’re applying. I don’t see this encouraged very often, but I strongly recommend that you email your potential supervisor and ask them if they might be interested in supervising such a project or if it fits into their wheelhouse. If all things go well, you might then ask if they could give feedback on it for the application. The most ideal situation is that you work together to craft the research proposal for your application (and for funding). This doesn’t always happen, but if you have the opportunity to work with a potential supervisor, who knows the ins and outs of the application process and department, this may give you an advantage. The last thing you want is a great research proposal with no supervisor to take you on. As far as potential supervisors are concerned, if you are able to work with a supervisor on your research proposal that might be a good indicator of how they are a supervisor. Just because someone is a top scholar in the field does not mean they will be a good supervisor. I suggest contacting current or past students to ask about their experiences and if they are willing to share them.
Your research proposal should state your research question upfront and as clearly as possible. You do not want to leave the admissions committee scratching their heads about what question you’re trying to answer. It should be a research question (singular) as too many questions might make it seem like you haven’t quite narrowed a clear trajectory for where you’re wanting to go. This trajectory will change when you start your course (and admissions committees know this) but they also want to see that you have the ability to articulate a focused project.
Current Debates. You cannot offer a full chapter literature review in your research proposal (nor should you). The skill of abbreviating the present state of a scholarly debate is a difficult one but important at this stage. Can you economically describe the issues, while mentioning the key players while still having room to write the other parts of your proposal?
Conversation Partners. What ancient and contemporary literature is going to be your major conversation partners? Be specific about what texts/evidence/archive you’re using and make sure that you’re engaging literature that is in other modern research languages relevant to the topic. At this point you might note your ability to work in these languages or any coursework you need to do to further your abilities. (Do not lie about your language abilities).
Contribution to knowledge. This is academic-speak for “why in the world this project matters.” What does this project contribute to the field, to our understanding of the sources, texts, theology, NT, etc. Be specific and be realistic about what your contribution will be. And don’t overstate it (avoid terms like “paradigm shift” in your language about the project). The contribution to knowledge allows you to showcase what is novel about your project and why it is important. If this work draws on previous research/publication (e.g. a master’s thesis) that you’ve done indicate how the two relate and how this furthers that previous research trajectory.
Method and methodology. What approach will you use to analyze the sources (conversation partners)? Why are these methods justifiable for the project? Think about the theoretical framework (e.g. disability studies, Paul within Judaism, etc.) but also about methodological framework (philology, reception history, comparative, etc.).
Offer a succinct but representative bibliography. Include just the key works that will you’ll be engaging with and drawing on. These are sometimes included in the word count, other times they are not (again, see the posted rules).
In addition to your potential supervisor, get as many people as you can from different/adjacent fields to give you feedback on your research proposal. Your past mentors, recommendees, senior colleagues you know, anyone. (If you’d like feedback from me, I’d be happy to help. Just give me a shout).
Understand also that it’s very likely that you’ll be doing different research proposals for different institutions! Not everyone has the same faculty who specialize in the same area with the same focus. Not every department has the same interests. So you have to be ready with multiple research proposals, or a research proposal that is flexible enough to jive with multiple institutions (just make sure that at application time you send the right proposal to the right institution).
Scholarships
Scholarships are very competitive and scarce in the UK. Sometimes they’re automatically considered and other times you have to apply for them separately. Some institutions have huge databases that can search funding between departments, colleges, external funders, etc. so pay attention to those and your eligibility! Major scholarships are likely to depend on your research proposal and potentially references. Often there are stages of acceptance, so you might notify your interest in the scholarship, you send in a research proposal. If it gets accepted you’re invited to a second round with revisions back from a committee. If they accept that one you might be a finalist with further revisions, etc. Keep in mind that while application deadlines can sometimes be open at UK institutions, scholarship deadlines are early (often around December-January), so again, pay attention to the posted deadlines!
What you’re doing with a research proposal is offering a proof of concept. You’re showing that this project is feasible. To be competitive for scholarships you need to show that your project is not only feasible but that the institution to which you are applying is the place to do it and that you are the scholar to do it. There is no magic formula that will guarantee you a scholarship. These are some principles that guide my scholarship/grant writing.
You want to make sure that the topic aligns perfectly with the institution, supervisor, resources available, trajectory of the department.
Try and relate your topic to something relevant in the current world. Contributions to knowledge that have a public facing element or implications wider than just the narrow specific subfield you’re working in might be more fundable. For me it’s not about being relevant or trendy for the sake of it. There is an ethical dimension to my research; I want it to actually help people outside of the academy. Find a way to work this into your research proposal.
Use non-specialist language. Not everyone on these committees is going to be an expert in your field, so you need to make sure you’re appealing with language shared across different fields or explaining and glossing your jargon along the way. It’s not always possible to write jargon-free, but it is a desirable aim.
Show how your project will contribute to the University. What seminars might benefit? What Institutes could you partner with? What resources are available here that are not available elsewhere?
That’s all I’m able to write at the moment, but I hope that this has been helpful. Do you feel like you have more to add? Leave a comment below. Good luck.