GitHub Copilot Chat is a tool that brings a chat interface to the editor, focusing on developer scenarios and natively integrating with VS Code and Visual Studio. It is built upon OpenAI and Microsoft’s work with ChatGPT and the new Bing. However, despite its promising premise, the current version of Copilot Chat leaves much to be desired.
A lot of the technical prowess from the team that incorporated GPT-4 into Bing Chat also allegedly went into GitHub Copilot Chat. Although, it’s hard to shake the feeling that Copilot Chat feels like it has some artificial brakes applied at the moment. Possibly due to its controlled and limited release.
One area where Copilot Chat excels is in explaining code blocks. The chatbot is adept at breaking down complex code structures, even when dealing with less-than-ideal code. This feature is beneficial in understanding and learning from one’s code. This feature is helpful from the perspective of developers taking over codebases they didn’t write. Many of us have been there where we’ve inherited terrible codebases.
However, when it comes to larger codebases with numerous dependencies and layers of abstraction, Copilot Chat’s lack of context can be a hindrance. While it performs well with essential functions and apps, it struggles to understand entire architectures or the nuances of an app’s coding style. This limitation is particularly evident when dealing with apps designed with separation of concerns, where components are “dumb”, and data fetching, storage, and other layers are abstracted. As a result, Copilot Chat sometimes fails to generate code that aligns with an app’s existing architecture and methodologies.
Ideally, Copilot Chat could index an entire codebase, providing more helpful suggestions that fit seamlessly within a project’s existing structure. Unfortunately, this functionality is not yet available. Nevertheless, being in limited release, it’s essential to recognise that the chatbot is still a work in progress, and improvements are likely on the horizon. An expanded scope would make Copilot so powerful and more useful.
In addition to its contextual limitations, Copilot Chat struggles with generating tests. The AI tends to duplicate existing tests and can produce incorrect tests due to its limited understanding. This shortcoming makes it more practical for developers to write tests manually, at least for now. It can sometimes learn from existing tests and help avoid repeating some patterns in test cases, but it’s not always that helpful. In one instance, I had a 3000-line test file I tried, and it struggled with it (due to the size and limited scope of Copilot GPT-4).
Lastly, the user interface could use improvement. The chat window is small, and a more user-friendly design would go a long way in enhancing the overall experience. It feels like something haphazardly bolted onto the IDE and not a proper integration. Hopefully, GitHub and Microsoft will work on making this better integrated and develop a UI that doesn’t feel so claustrophobic.
In conclusion, GitHub Copilot Chat is a promising tool that requires refinement before it can revolutionise the coding landscape. Its ability to explain code blocks has proven to be a valuable resource in some respects. Still, it needs to overcome its contextual limitations and improve its user interface to become an indispensable asset for developers. As it stands, Copilot Chat is a fascinating glimpse into the future of AI-assisted coding, but it has yet to realise its potential fully.
