Thoughts on GitHub Copilot

About a month ago, I decided to turn on GitHub Copilot in Visual Studio Code and see what the latest version of the AI pair programmer was all about. I wasn’t quite sure what to expect. I’ve played with ChatGPT a little bit but found it more challenging to think of things to ask about than I expected. And whenever I did ask, I always felt underwhelmed by the response. So my expectations were set appropriately low for GitHub Copilot.

Copilot integrates with Visual Studio Code in three ways.

• Super autocomplete - this is what I call the automated suggestions that pop into the editor as you’re typing.
• Inline chat - you can select some code and hit Ctrl+I to enter a prompt related to the highlighted code.
• Chat window - the sidebar screen that allows you to have ChatGPT-like conversations with Copilot.

I break down these three integration points because I had very different experiences with each of them.

Super autocomplete. At first I was disappointed, as I wasn’t really getting any suggestions when doing some ESLint tasks. And I don’t mean I didn’t get good suggestions, I mean got zero suggestions. I’m not sure if that’s because what I was doing was too complex or too specialized, but most of the time I forgot Copilot was enabled. That all changed when I started a brand new project. Suddenly, Copilot seemed like it was reading my mind. Not only could it anticipate the name of the next method I wanted to write, it often correctly guessed the arguments, and sometimes, could even fill in the method body with about 80% accuracy. Additionally, it was excellent at generating JSDoc descriptions for the methods I just wrote. I estimate I probably got about twice as much done on this new project with Copilot than I would have without it. Even incorrect suggestions were often close enough to reduce my typing by about 50%. If you have any kind of physical disability or injury, those saved keystrokes can really add up.

Inline chat. My positive experience with super autocomplete made me excited to start using inline chat to see what else I could accomplish. Here, though, my experience was quite different. It seemed that no matter what I typed, there was always something wrong. More frequently than not, the inline prompts generated invalid syntax. When the syntax was valid, there was almost always some inexplicable style difference from the highlighted code (using single quotes instead of double, quoting object literal properties when they weren’t before). One time, it actually deleted most of my code and replaced it with a comment that said “...the rest of the code.” I eventually gave up on asking Copilot to refactor code, falling back to regular expression find and replace. The /doc prompt seems to work the best and was able to generate useful JSDoc.

Chat window. Chatting with Copilot was hit or miss. I found it exceptional for explaining things, which cut down on the number of times I jumped onto Google to find an answer for something. (When you use the /explain prompt in the inline chat box, it will open the chat window.) What it didn’t do particularly well was generate code from scratch. I asked it to generate unit tests for a new class I had just written, and I explicitly said I wanted it to use the built-in Node.js test runner and asserts. It output tests designed for use with Jest, but more importantly, there were two big problems with the code:

1. It got the method names wrong. The words in the method names were sometimes in the opposite order from the code. For instance, I had a method called implSet but Copilot generated a test for a method called setImpl.
2. It generated tests that tried to access private class fields as part of its assertions. Effectively, these tests were useless.

Overall, I definitely feel that AI-assisted programming will be the way we are all writing code in a short amount of time. Further, this is going to be the way children are taught to code, with an AI pair programmer always at their side to help out. But for right now, the code generation outside of super autocomplete is way too flaky for me to use on any kind of regular basis. I’ll continue to use the /explain and /doc prompts, and keep super autocomplete enabled, but I probably won’t attempt any code generation from scratch or refactoring for a while.

How has your experience been with GitHub Copilot? Reply to this email and let me know!

​

Key Takeaways

• GitHub Copilot can be hit or miss, depending on how you're using it.
• Super autocomplete seems like the most polished part of GitHub Copilot, and easily the part I found most useful.
• Generating code from scratch was usually a miss, with multiple problems or inconsistencies.

More about AI

📚 The Age of AI by Henry A. Kissinger, Eric Schmidt, Daniel Huttenlocher
This book was released far before ChatGPT became an everyday discussion point and digs into what integrating AI into our society really means.

📝 Getting started with GitHub Copilot​
A gentle introduction to setting up and using Copilot. Full of helpful tips that I didn't know before I found this page.

🎬 The "NEW" AI podcasting tool that saves hours by Pat Flynn
If you're looking for more ways that AI is infiltrating the real world, this video shows how it can be used to quickly and easily edit both audio and video podcasts.

🤝 1-on-1 Coaching for Software Engineers. I work with senior-level software engineers (and above) to work through the difficult parts of your job. Whether that be leading without authority, improving your productivity, achieving work-life balance, or convincing your VP of your idea, I can help you achieve your goals.

Stuff I've Enjoyed This Month

🎬 TypeScript Origins: The Documentary by OfferZen Origins
This full-length, free documentary interviews all of the key players in TypeScript's history and retraces its origins. Very interesting background.

📝 Prettier's CLI: A Performance Deep Dive by Fabio Spampinato
Prettier just went through an experiment with creating a Rust-compatible formatter, and the result was actually to improve the performance of the JavaScript CLI. This post details how they did it.

🎬 Bun 1.0 is here by Bun
Bun, the JavaScriptCore-based alternative to Node.js, just hit 1.0. This short video gives you an overview of the runtime and how it compares to Node.js.

💻 Pagefind by CloudCannon
Pagefind is a static search generator that you can embed on your statically-generated website to perform search quickly and locally, allowing complete control over the UI.

🎬 A simple way to learn complex skills by Benjamin Keep, Phd, JD
A complex skill is something that requires you to do multiple things at once. This video shows a proven technique for learning complex skills by rotating your focus.

What I'm Working On

🏠 Real Estate: We finally heard back from the insurance company on the additional claims we put in for the hail damage, and we got most of them approved. Final work will be starting this week and I'll be glad to move on to other challenges. Follow my Instagram for photos.

📝 Open Source: I started working on a new project that I'm excited about. It's a small utility library, but I think it solves some big problems. I'm not ready to share quite yet, so stay tuned!

💻 ESLint: The work for ESLint v9.0.0 has begun, and we are likely looking at an alpha release in the next two months. You can follow the progress on our project board.