Something interesting has happened with the famed GPT-4 model from OpenAI lately, and it’s not just me that has noticed. Many people have been talking about how GPT-4 lately feels broken. Some say it’s nerfed, and others are saying it’s possibly just broken due to resource constraints. There was a discussion recently on Hacker News in this thread which received 739 comments.
All signs indicated that OpenAI had changed something significant with ChatGPT lately and its GPT-4 model. Users reported that questions relating to code problems were producing generic and unhelpful answers.
An OpenAI employee on Twitter claims that all models have been static since March 2023.
Notice the wording? Logan says the API does not change, and the models are static. The noticeable reduction in quality people report is with ChatGPT, not the GPT-4 API directly. That is still producing desired results.
I tend to use the API for a lot of my prompting. I use the Chatbot UI app and plug in my GPT-4 API key. I still use ChatGPT to keep my API costs down. And I have noticed the quality of the answers has changed. While the models might not have changed, the context window appears to have changed. So, even though some complaints seem to be about ChatGPT, the API is acting differently, too. I’ve got an app with the temperature set to zero that has started acting weird recently.
ChatGPT is doing things I haven’t seen it do before with GPT-4 like correct itself mid-generation (which is cool), and the speed for ChatGPT with GPT-4 is noticeably faster (almost as fast as 3.5-turbo from my observations). OpenAI appears to be messing with the middleware and parameters and not being honest about it.
It’s apparent that maybe OpenAI is paying the alignment tax. The more they try and make it safer and faster, the worse the results get.
