How ChatGPT could replace IT network engineers

Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More

• Configuration management
• Troubleshooting
• Documentation

I then developed a set of prompts — admittedly not optimized — to determine whether or not the tool could, in fact, be an asset to network engineers in one or more of these areas.

Configuration management

To test ChatGPT’s ability to add value in configuration management, I submitted the following prompts:

• Can you generate a complete example configuration for a Cisco router with the purpose of starting an internet exchange from scratch?

• What about Juniper?

• Can you create a Jinja template for each vendor?

The ChatGPT results are extensive, so space — and my respect for the boredom limits of those reading this — limits an exhaustive reproduction of them here, but I have posted the complete transcript of all of the ChatGPT prompts and results on GitHub for those searching for a non-pharmaceutical substitute for Ambien.

So, in the case of configuration management, ChatGPT performed fairly well on basic configuration tasks, and I concluded that it is aware of vendor-specific syntax and can generate configurations. However, the configurations generated by the system should be carefully inspected for accuracy. The generic prompts I tested would be akin to building a quick lab, a task most young networking engineers find tiresome at a minimum and clearly a chore that can be handled by the technology (with, again, some human oversight).

Troubleshooting

To test ChatGPT’s prowess at troubleshooting network engineering challenges, I turned to Reddit, and specifically the /r/networking subreddit to find real-world questions posed by network engineers to their peers. I pulled a few questions from the thread and proposed them to ChatGPT without optimizing the prompt, and the chatbot handled the easier questions well, while it struggled with the more difficult challenges.

Notably, I specifically asked a question that required knowledge of STP, or the Spanning Tree Protocol, a switch capability responsible for identifying redundant links that could result in unwanted loops. Frankly, my opinion is that ChatGPT understands STP better than many networking professionals I’ve interviewed over the years.

At present, ChatGPT can’t replace experienced networking professionals for even slightly complex issues, but it wouldn’t be alarmist to suggest that it might result in the obsolescence of many subreddits and Stack Overflow threads in the coming years.

Automating documentation

This was the area of highest deficiency for ChatGPT. The chatbot initially assured me that it could generate networking diagrams. Knowing it is a text-based tool, I was obviously skeptical, a prejudice that was confirmed when I asked it to generate a diagram and it explained to me that it doesn’t have graphical capability.

Further prompting for network documentation led to the realization — confirmed by ChatGPT — that I needed to provide a detailed network description for it to provide a network description, clearly not a value add. Thus, in the case of automating documentation, the chatbot not only failed, but was guilty of generating lies and deception (so perhaps it’s closer to demonstrating human characteristics than we think). In fairness to AI in general, there are AI applications capable of generating images, and it’s very possible one of those may be capable of producing a usable network diagram.

I then asked ChatGPT if it could generate a network description based on a router configuration file, and it provided a decent summary of what’s configured until it apparently reached the limits of its computational commitment to my prompt, a limit likely implemented by its designers. It is, after all, a free tool, and resources are expensive, especially for an organization burning meaningful cash these days.

Conclusions

A few of the challenges I encountered in my brief experiment when using ChatGPT for network engineering include:

• Ensuring accuracy and consistency
• Handling edge cases and exceptions
• Integration with existing systems and processes

My guess is these issues are not unique either to ChatGPT or AI applications generally, and some cursory research may explain why. Cornell researchers have been studying large language models (LLMs) for some time and “draw a distinction between formal competence — the knowledge of linguistic rules and patterns — and functional competence, a set of skills required to use language in real-world situations.”

Also from some of their research summaries: “Too often, people mistake coherent text generation for thought or even sentience. We call this a “good at language = good at thought” fallacy. Similarly, criticisms directed at LLMs center on their inability to think (or do math or maintain a coherent worldview) and sometimes overlook their impressive advances in language learning. We call this a “bad at thought = bad at language” fallacy.

This analysis is consistent with my experience preparing this article: Specificity reigns supreme when it comes to putting ChatGPT to work. Large, open-ended prompts on complex topics highlight a lack of “functional competence” in the chatbot, but that reality doesn’t neutralize its impressive capabilities when employed for specific tasks by an individual skilled in using it properly.

So, can ChatGPT replace network engineers?

Not yet.

Mike Starr is the CEO and founder of trackd.