Blog Home

ChatGPT: Initial Thoughts

By Robert Alexander

2023-03-26

I've been seeing lots of buzz about ChatGPT

I'm wary to invest too much of my time chasing after technology that doesn't help me in some way, so I try to avoid fads and I'm not an early adopter. There's lots of great work to be done with the tools we have, and lots of failed projects that hitch their wagon to marketing fluff. Cutting through the fluff is incredibly valuable, but challenging.

Lots of content I've read previous to this suggested that:

For most of that, I suspect it's too soon to tell. I'm not an early adopter, so "wait and see" is a great strategy. But I'd like to jump on-board as soon as the value is clear.

Many online comments focused on tricking ChatGPT into making a mistake, or showing how clever the author was. Those comments usually aren't helpful, as they don't capture the utility. A useful tech doesn't need to be all good; many good tools have sharp edges.

Before I take a look, I want to share my prior thoughts on these concerns.

On jobs: technology regularly displaces workers. It's silly to expect that over a long enough time period any specific job, skills, and way of working will continue to exist. I expect from the start, that if ChatGPT does anything useful that it has the ability to disrupt workers. That change could be fairly rapid, but it could also be more gradual. On a personal level, I'm biased, and what I care most about is if my job will disappear before I retire. If ChatGPT ends up changing how we work, that's OK, but I'd like to learn it before my skills grow stale.

A key question for me is what society could do with extra productivity gained from AI written software. Maybe there's unmet demand for software, and we'll mostly see more software production. Maybe there's not enough demand to support current salaries, and we'll see pay fall. Maybe there's so much productivity gain that people who don't use AI to write software will get edged out. Maybe it's so easy to use ChatGPT that project managers don't need engineers any longer, and the entire profession shrinks.

On the rate of improvement: as an outsider, it's hard to measure the progress. I expect hype to obscure actual progress for a while, the marketing budget is clearly quite large already. I don't have a prior stance on this, but I intend to watch the progress.

The comment that really encouraged me to take a look mentioned that the author "found ChatGPT to be useful multiple times a day, and that their usage was increasing". That stickiness and increased usage is interesting.

The key question I'm trying to figure out is if ChatGPT is currently productive for some use case I care about, or if I expect it to be productive in a short time period. If not, I'm happy to check in again at a later time.

So let's get started. Creating an account on chat.openai.com is quick although you've got to validate both an email address and phone number. You'll see some prompts about the project being for research, but you're quickly dropped to a chat window. That's a decent on-boarding.

Over the next couple days I bounced some questions over to ChatGPT to see what it could do. Here's some of the conversations and my thoughts. I've tried to include all the prompts, but I've indicated where I've trimmed the logs.

Private CAs

How do I create a private CA?

Creating a private Certificate Authority (CA) involves a few steps:

Choose a Root CA: First, you need to choose a root CA that will issue digital certificates to your organization. You can use open-source software like OpenSSL or buy a commercial CA.

Create the Root Certificate: Once you have chosen a root CA, you need to create the root certificate. The root certificate is the first certificate that your CA will issue, and it is used to sign other certificates. The root certificate should be kept secure, as it is the foundation of trust for all other certificates in your organization.

Generate Intermediate Certificates: To create a hierarchy of trust, you can create intermediate certificates that are signed by the root certificate. These intermediate certificates can be used to issue other certificates, such as server certificates or client certificates.

Issue Certificates: With your CA infrastructure in place, you can now issue certificates to your organization. You can issue certificates for servers, clients, and other devices.

Maintain the CA: As with any security infrastructure, you need to maintain your CA to ensure that it remains secure. This includes updating software, revoking certificates, and monitoring for security breaches.

Overall, setting up a private CA involves a significant amount of technical expertise, and it is important to ensure that you have the necessary skills and knowledge to do so. It is also recommended to seek expert advice and follow best practices to ensure the security and reliability of your CA.

The reply was much longer than I expected, which felt like a negative. ChatGPT can summarize though, so I asked again.

In short, how do I create a private CA?

Creating a private CA (Certificate Authority) involves the following steps:

Choose and set up a server: You'll need to choose a server to act as your CA, and set it up with a secure operating system, such as Linux.

Install OpenSSL: OpenSSL is an open-source toolkit that provides a way to create and manage digital certificates. You'll need to install it on your server.

Generate the CA key: You'll use OpenSSL to generate a private key for your CA. This key will be used to sign and issue certificates.

Create the CA certificate: Using OpenSSL, you'll create a self-signed certificate for your CA. This certificate will identify your CA and be used to sign other certificates.

Configure your applications: Once you've created your CA and its certificate, you'll need to configure your applications to trust your CA. This involves importing the CA certificate into the trusted certificate store for each application.

Issue certificates: With your private CA set up and trusted, you can issue certificates for your servers and services as needed. These certificates will be signed by your CA and trusted by your applications.

Note that setting up a private CA can be complex and requires careful attention to security. It's important to follow best practices for securing your server, managing your keys and certificates, and protecting against unauthorized access.

Hmm. That wasn't really shorter and it's different guidance. We've lost mention of intermediate certificates and options other than OpenSSL. We've gained a server, Linux, and trust stores. I don't think either answer is wrong, but the change of details was unexpected.

You've got to prompt for a summary differently:

Can you summarize that?

To create a private CA, you need to set up a server, install OpenSSL, generate a CA key, create a self-signed CA certificate, configure your applications to trust your CA, and issue certificates as needed. It's important to follow best practices for security and key management.

Great. It's interesting that it continues to include the warning about security. This makes sense to me, as we've got to worry about AI telling people to do dangerous things. The disclaimer to "follow best practices" is already starting to feel redundant, although it's never bad advice.

This doesn't feel very useful to me though. I've got to drill down further to get at which OpenSSL commands to run. If I didn't see the first response, I may not realize I should use intermediate certificates. It sort of feels like ChatGPT needed to fit the reply into X characters, and it left off various parts to make it fit. Ideally, I want the answer to be exactly as long as needed, but no longer. That's a high bar, but ChatGPT needs to be more useful than DuckDuckGo (DDG, my preferred search engine) and Google (my fallback), otherwise it's not useful to me for this task.

What kind of answers do I get when I search DDG? Here I'll use the shorter "create private CA" query, as I know search engines don't need the fluff.

Right off the bat there's helpful content.

So I feel like DDG wins this round, but if I stick with ChatGPT can I get an equivalent experience? It looks like I'll need to directly ask for alternatives, background, and the specific commands to run.

What alternatives should I consider instead of creating my own private CA?

If you're considering alternatives to creating your own private CA, here are some options:

Public CA: You can obtain digital certificates from a trusted public CA such as Let's Encrypt, DigiCert, or Comodo. This can be a good option if you only need a few certificates, and don't want to manage your own CA.

Managed PKI service: You can also use a managed PKI (Public Key Infrastructure) service, which provides the benefits of a private CA without the hassle of managing it yourself. Some popular managed PKI services include GlobalSign, Entrust, and Symantec.

Certificate management tools: You can use certificate management tools such as Hashicorp Vault, Venafi, or Sectigo Certificate Manager to automate the issuance and renewal of certificates and manage them centrally.

Self-signed certificates: In some cases, self-signed certificates can be used as an alternative to a private CA. However, self-signed certificates are not trusted by default and require manual distribution of the certificate to all parties that need to trust it.

It's important to consider your specific use case, security requirements, and budget when choosing an alternative to creating your own private CA.

Here I'm tempted to say that the remark about certificate management tools is technically wrong as those help me manage certificates, but I've still got to create them. Overall, that remark is helpful as it reminds that there's tooling out there to support me, so I'd say this works.

Let's paste in one of the section headers from the OpenSSL blog post to see if we can get background info.

How do CAโ€™s and certificates, basically, work?

CA (Certificate Authority) and digital certificates work together to establish trust in online communication. Here is a brief overview of how they work:

A website or server that wants to establish a secure connection with a client generates a public/private key pair.

The website/server sends its public key to a trusted CA along with some identifying information, such as the website domain name, organization name, and location.

The CA verifies the identity of the website/server and issues a digital certificate that includes the public key and the identifying information.

When a client (such as a web browser) connects to the website/server, it receives the digital certificate and verifies its authenticity by checking that it was issued by a trusted CA and that it has not been tampered with.

If the certificate is valid, the client and the website/server establish a secure connection using encryption, ensuring that data exchanged between them is private and secure.

The process of verifying the identity of the website/server and issuing a certificate is known as the certificate issuance process. The certificate issuance process is crucial to the security of the digital certificate ecosystem, and it is why it is essential to have trusted CAs. The CA serves as a trusted third party that verifies the identity of the website/server and issues the certificate.

It's worth noting that the trust in a digital certificate is only as strong as the trust in the CA that issued it. Therefore, it's essential to use reputable and trusted CAs and to keep the private key used to sign the certificates secure.

This background info has some concerns. It's very casually written and vague on the details. None of this feels wrong though. Can we press for more information about how verification works?

How does Let's Encrypt verify identity prior to issuing a certificate?

Let's Encrypt verifies the identity of the domain owner before issuing a certificate using an automated process called the Automatic Certificate Management Environment (ACME) protocol. Here's how it works:

Let's Encrypt verifies that the domain requesting a certificate is owned by the party requesting it. This is typically done through a challenge-response mechanism, where Let's Encrypt sends a challenge to the domain that must be responded to correctly.

Let's Encrypt checks that the requester has control over the domain's DNS or HTTP server by having them add a specific DNS record or file to their server.

Let's Encrypt verifies that the domain name is not on any blacklist of known malicious domains.

Let's Encrypt uses public databases to check that the requester's contact information matches the WHOIS registration for the domain.

If all of these checks pass, Let's Encrypt will issue a certificate for the requested domain. The process is automated, which makes it much easier and faster to obtain a certificate from Let's Encrypt than from traditional CA's, which typically require manual verification of identity.

It's worth noting that Let's Encrypt's validation process is optimized for domain validation only, and it does not provide any extended validation or organization validation features that some other CAs offer. However, Let's Encrypt's approach has helped to make SSL/TLS encryption more accessible to a wider range of website owners.

This answer is wrong. Let's Encrypt does not check WHOIS. We can find discussions about this and why it wouldn't work. ChatGPT can hallucinate facts, which appears to be happening here.

I wasn't expecting to see a wrong answer so early. I figured that the incorrect ChatGPT responses I was seeing were cherry-picked. For me, a wrong answer is a big problem. I've got a pretty decent ability to tell if a blog post is written by someone who knows what they are talking about. ChatGPT is hard to read, this answer is written in an authoritative way.

Let's press on that incorrect info.

Does Let's Encrypt use WHOIS information when validating domain ownership and issuing a certificate?

Let's Encrypt does not rely on WHOIS information to validate domain ownership and issue certificates. Let's Encrypt's automated validation process uses a combination of DNS and HTTP-based challenges to verify that the domain owner has control over the domain.

When a certificate is requested, Let's Encrypt generates a challenge that the domain owner must complete to prove that they control the domain. For example, Let's Encrypt may ask the domain owner to add a specific DNS record or file to the domain's DNS or web server. This process is designed to be automated and does not rely on manual verification of WHOIS information.

However, Let's Encrypt may use WHOIS information to check for obvious mismatches between the requester's contact information and the WHOIS information for the domain. If a mismatch is detected, Let's Encrypt may require additional validation steps or deny the certificate request.

It's important to note that while Let's Encrypt does not rely on WHOIS information for domain validation, WHOIS data is still publicly available and can be used by attackers for social engineering and phishing attacks. Therefore, it's important for domain owners to keep their WHOIS information up-to-date and accurate to prevent such attacks.

Wow... This is a confusing answer, it doesn't appear internally consistent. First is says they don't use WHOIS, then it says they may use it to deny a request. I clicked regenerate and it made a similar claim that also seemed internally inconsistent.

My traditional approach of searching DDG is also subject to wrong answers, but I have existing tools to sense if the information I'm reading is wrong and I can get diverse second opinions. I'm sure there's some queries where the entire first page of search results have wrong answers, but ChatGPT just gave me a wrong answer twice in a row and the first link of the first page tells me ChatGPT is wrong.

This is a deal breaker for me, for this sort of question. I get negative value if I'm trying to answer a question and I'm given incorrect answers. Thankfully I was able to fall back to DDG and verify that the answer was wrong, but I only did that because I suspected an issue. If I only fact check some responses, then I should expect to be lied to sometimes. If I end up running all responses through DDG, what have I really gained?

Is this fixable? I didn't see a way to re-engineer my prompt to get ChatGPT to admit its mistake. Even if I did, would I need to do that for every prompt? That really cuts down on the usability, especially when ChatGPT is giving vague, long winded answers. For this specific prompt, I flagged the answer as wrong, so presumably ChatGPT will learn. But to replace a search engine, I need to to give correct answers even when I don't know the actual answer. This is a "wait and see" problem for me.

Law

I won't include the prompts here, and I'm not a lawyer, but I asked a couple legal questions and got very questionable answers. Given the concerns seen earlier, I wouldn't think about using ChatGPT as a lawer, but further evidence of issues was troubling.

Stress

What are good ways to reduce stress?

There are many ways to reduce stress, and what works for one person may not work for another. Here are a few effective ways to reduce stress:

Exercise: Exercise is a great way to reduce stress and improve mood. It releases endorphins, which are natural feel-good chemicals that can help reduce feelings of stress and anxiety.

Meditation: Meditation is a technique that involves focusing the mind on a particular object, thought, or activity to achieve a calm and relaxed state. Regular meditation practice can help reduce stress and anxiety.

Deep breathing: Deep breathing exercises can help reduce stress and promote relaxation by slowing down the heart rate and lowering blood pressure. Taking a few minutes each day to practice deep breathing can help you feel more calm and centered.

Get enough sleep: Getting enough sleep is essential for reducing stress and maintaining good mental and physical health. Aim for 7-9 hours of sleep each night.

Connect with others: Spending time with friends and family, or participating in activities that you enjoy, can help reduce stress and improve your mood.

Time management: Poor time management can contribute to feelings of stress and overwhelm. By prioritizing tasks, setting realistic goals, and breaking larger tasks into smaller, more manageable ones, you can reduce stress and increase productivity.

Remember, reducing stress is a process that takes time and practice. Try incorporating one or more of these strategies into your daily routine and see what works best for you.

This reply seems helpful, but isn't above what any random blog post will mention. I ran a couple similar questions about different topics and while they all seemed OK to me, they were shallow in content. I'm not seeing anything pulling me off DDG.

If I was running a blog spam website, I'd take a longer look at ChatGPT. I suspect many of those sites are already using copy/pasted content with simple modifications already, so I don't think ChatGPT is a revolution in that space. It may be a little cheaper to expand the content to cover more topics, but this seems like an incremental improvement only. That's probably "bad for society" as we may continue to see the internet fill with low value content, but I don't see this triggering a massive shift.

If I was a BuzzFeed author, I may look at ChatGPT to generate ideas. But looking at some BuzzFeed articles about reducing stress, BuzzFeed seems like better content. I don't see BuzzFeed outsourcing articles to ChatGPT fully, in the near future.

Fitness

I'm a beginner spinner. What's a good workout for me to burn some fat? I have 20 minutes to work out.

As a beginner spinner, it's important to start with a workout that is manageable and achievable for your fitness level. Here is a 20-minute spinning workout that can help you burn some fat:

Warm-up (5 minutes):
Start with a light resistance and pedal at a moderate pace to get your body warmed up and your heart rate up.

Main Workout (15 minutes):
For the main workout, alternate between periods of high-intensity effort and recovery periods. Here's an example:

Sprint: Increase resistance and pedal as fast as you can for 20 seconds.
Recovery: Lower resistance and pedal at a moderate pace for 40 seconds.
Hill climb: Increase resistance and pedal at a slower pace for 60 seconds.
Recovery: Lower resistance and pedal at a moderate pace for 40 seconds.

Repeat the above cycle four times, for a total of four minutes of high-intensity effort and four minutes of recovery.

Cooldown (5 minutes):
Finish with a light resistance and pedal at a slow pace for five minutes to cool down and bring your heart rate back to normal.

Remember to listen to your body and adjust the resistance and intensity as needed. As you become more comfortable with spinning, you can increase the intensity and duration of your workouts. Also, don't forget to stretch after your workout to help prevent injury and improve flexibility.

Meh. Ignoring the multiple math errors in the timing, this doesn't do much for me. A similar DDG search finds lots of videos and workouts paired with Spotify playlists.

Children's books

Write a children's bedtime story about a cat, using words a first grade student can read easily.

Once upon a time, there was a fluffy, gray cat named Mittens. Mittens was a very special cat because she had a magic collar that made her meow like a person!

One day, Mittens went on an adventure through the neighborhood. She meowed hello to all the friendly dogs and waved her paw at the squirrels in the trees.

But as the sun started to set, Mittens realized she was lost. She meowed and meowed, but nobody seemed to hear her.

Just when she was starting to feel very scared, she saw a little girl named Lily walking down the street. Mittens ran up to Lily and meowed, "Help me, I'm lost!"

Lily picked up Mittens and took her home. She gave Mittens a cozy bed to sleep in and a big bowl of cat food to eat. Mittens was so grateful to Lily for helping her find her way home.

From that day on, Mittens and Lily became the best of friends. Mittens would often visit Lily's house to play and cuddle.

And every night, before Mittens went to sleep, she would meow a special thank-you to Lily for being such a kind and helpful friend. And with that, Mittens would curl up in her cozy bed and fall fast asleep.

I think this was the best result I got so far. It has a friendly protagonist, some magic, a problem, a resolution, a hero, and a moral. The book is a little boring and obvious, but most children's books aren't great works of art. The magic meow is a little strange, the story reads the same if that element is removed, so it feels out of place to me. I'm a little unsure if Lily brought the cat to the cat's home or to her own home. That worried me a little, was Mittens kidnapped? An editor may help clean this stuff up, human writers often need outside opinions, so it doesn't seem fair to overly criticise a first draft. The reading level seems about right, which is great.

I think I'll try asking ChatGPT for some more children's stories. When you're helping a child read, getting them a new book helps them process the words they are seeing instead of relying on memory. Being able to generate lots of short stories is helpful. Ideally, I'd like the story to be interesting so we can bond while reading, enjoy the process, and feel eager to read more. I'm going to try asking ChatGPT to generate stories based on my child's ideas.

On the jobs front, should writers be concerned? I'm not sure, I'd explore this more if I was a writer. Simply writing text isn't the hard part, it's trying to build a cohesive and interesting story. Maybe we can engineer prompts to get cohesion? This still seems like an incremental improvement an author would use for inspiration. Authors already have tools to help them get inspired, this seems like another one.

Trivia

I asked a couple trivia questions, and most looked like correct answers. But ChatGPT has insidious errors sometimes, like this one about moons:

How many moons are there in our solar system?

As of my knowledge cutoff date of 2021, there were 214 confirmed moons orbiting the planets in our Solar System. Here is a breakdown by planet:

Mercury and Venus do not have any moons.

Earth has one moon, which is simply called "the Moon."

Mars has two small moons, Phobos and Deimos.

Jupiter has 79 confirmed moons, including the four largest ones: Io, Europa, Ganymede, and Callisto.

Saturn has 82 confirmed moons, including the largest one, Titan.

Uranus has 27 confirmed moons, including the largest ones, Titania and Oberon.

Neptune has 14 confirmed moons, including the largest one, Triton.

Pluto, which is now classified as a "dwarf planet," has five known moons: Charon, Styx, Nix, Kerberos, and Hydra.

It's possible that new moons may be discovered in the future, as our technology improves and we continue to explore our Solar System.

This is an impressive answer. We're going from "solar system" to "planets and dwarf planets" to moon counts, to a summation. I didn't fact check each planet's answer, but I noticed that 1 + 2 + 79 + 82 + 27 + 14 + 5 is actually 210. That's easy to miss as usually computers are good at math, it's unexpected! Let's ask again:

That math seems incorrect, are there other moons you didn't mention?

I apologize for any confusion. I may have misspoken in my previous response. As of my knowledge cutoff date of 2021, there were over 200 confirmed moons in our Solar System. Here is a more detailed breakdown by planet:

Mercury and Venus have no moons.

Earth has one moon.

Mars has two moons, Phobos and Deimos.

Jupiter has 79 known moons, including the four largest ones: Io, Europa, Ganymede, and Callisto.

Saturn has 82 known moons, including the largest one, Titan.

Uranus has 27 known moons, including the largest ones, Titania and Oberon.

Neptune has 14 known moons, including the largest one, Triton.

Pluto, which is classified as a dwarf planet, has five known moons: Charon, Styx, Nix, Kerberos, and Hydra.

So, the total number of known moons in our Solar System as of my knowledge cutoff date was 211. It's possible that new moons may be discovered in the future as our technology and techniques for discovery improve.

211? Yikes.

I asked one more time and it claimed 205 moons this time. ChatGPT reports the same number of moons per planet each time, but adds them up to a different wrong number. I suppose this makes sense, ChatGPT isn't a math model, it's a language model.

I'm very wary of answers it gives to trivia questions.

DevOps scripts

As a programmer, I care most about software. Let's take a look.

Here's a short script I needed to write recently. It's a hold over until I got a chance to improve some automation. Let's see how well ChatGPT can do with this task.

Write a bash script that can lookup alexsci.com and alert when the IP address changes

Here's an example bash script that uses dig command to lookup the IP address of alexsci.com and alerts if it changes:

#!/bin/bash

# Set the domain name to lookup
DOMAIN="alexsci.com"

# Set the initial IP address
IP=""

while true; do
  # Lookup the IP address using dig command
  NEW_IP=$(dig +short $DOMAIN)

  # Compare the new IP address with the previous one
  if [[ "$IP" != "$NEW_IP" ]]; then
    # Alert if the IP address has changed
    echo "The IP address of $DOMAIN has changed from $IP to $NEW_IP"

    # Update the IP address
    IP="$NEW_IP"
  fi

  # Wait for 1 minute before checking again
  sleep 60
done
          

To run this script, save it to a file (e.g. ip_monitor.sh), make it executable using the command chmod +x ip_monitor.sh, and then run it using the command ./ip_monitor.sh. The script will run indefinitely, periodically checking for changes in the IP address of alexsci.com and alerting when there is a change. You can stop the script at any time by pressing Ctrl+C.

This is a helpful starting place, but has issues. The output of dig isn't sorted, so I see an alert basically every loop. Can we fix it?

Can you change that code to handle multiple A records in the DNS response?

Sure! Here's an updated version of the script that handles multiple A records in the DNS response:

#!/bin/bash

# Set the domain name to lookup
DOMAIN="alexsci.com"

# Set the initial IP addresses to an empty array
IP=()

while true; do
  # Lookup the IP addresses using dig command and store them in an array
  NEW_IP=($(dig +short $DOMAIN | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'))

  # Compare the new IP addresses with the previous ones
  if ! cmp -s <(printf '%s\n' "${IP[@]}") <(printf '%s\n' "${NEW_IP[@]}"); then
             # Alert if the IP addresses have changed
             echo "The IP addresses of $DOMAIN have changed from ${IP[@]} to ${NEW_IP[@]}"

             # Update the IP addresses
             IP=("${NEW_IP[@]}")
             fi

             # Wait for 1 minute before checking again
             sleep 60
             done
          

In this updated script, we use dig command to lookup the IP addresses of the domain name and use grep to filter out only the IPv4 addresses. We then store the IP addresses in an array and compare the new array with the previous one using cmp. If the two arrays are not identical, we alert that the IP addresses have changed and update the IP array with the new IP addresses.

Nope, that puts the addresses in an array, but doesn't sort them or otherwise address an ordering problem. That's an important detail that as a developer I noticed, but ChatGPT did not. I was also hoping to run the script via cronjob, so the loop isn't needed. Let's keep trying.

Can you update the script to handle DNS responses that don't use the same order each time and can be run as a one-shot from a cron job?

[snip]

#!/bin/bash

# Set the domain name to lookup
DOMAIN="alexsci.com"

# Set the initial IP addresses to an empty array
IP=()

# Lookup the IP addresses using dig command and store them in an array
NEW_IP=($(dig +short $DOMAIN | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'))

# If the IP array is empty, set the current IP addresses as the initial ones
if [ ${#IP[@]} -eq 0 ]; then
  IP=("${NEW_IP[@]}")
fi

# Compare the new IP addresses with the previous ones
if ! cmp -s <(printf '%s\n' "${IP[@]}" | sort) <(printf '%s\n' "${NEW_IP[@]}" | sort); then
             # Alert if the IP addresses have changed
             echo "The IP addresses of $DOMAIN have changed from ${IP[@]} to ${NEW_IP[@]}"

             # Update the IP addresses
             IP=("${NEW_IP[@]}")
             fi

[snip]

Those sort commands fix the first bug. However, the one-shot program always fails since IP starts as an empty array on every loop. ChatGPT doesn't notice that it should persist the IP addresses somewhere else. Let's keep trying.

Can you update the script to handle DNS responses that don't use the same order each time and can be run as a one-shot from a cron job?

[snip]

#!/bin/bash

# Set the domain name to lookup
DOMAIN="alexsci.com"

# Set the file path to store the IP addresses
FILE="/path/to/ip_addresses.txt"

# Create the file if it doesn't exist
touch $FILE

# Read the previous IP addresses from the file
IP=($(cat $FILE))

# Lookup the current IP addresses using dig command and store them in an array
NEW_IP=($(dig +short $DOMAIN | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'))

# If the IP array is empty, set the current IP addresses as the initial ones
if [ ${#IP[@]} -eq 0 ]; then
  IP=("${NEW_IP[@]}")
fi

# Compare the new IP addresses with the previous ones
if ! cmp -s <(printf '%s\n' "${IP[@]}" | sort) <(printf '%s\n' "${NEW_IP[@]}" | sort); then
             # Alert if the IP addresses have changed
             echo "The IP addresses of $DOMAIN have changed from ${IP[@]} to ${NEW_IP[@]}"

             # Update the IP addresses
             IP=("${NEW_IP[@]}")

             # Write the new IP addresses to the file
             echo "${IP[@]}" > $FILE
fi

[snip]

This still doesn't work. It sets the "current IP address" to empty and then to the "new IP address" which causes the comparison to always match. At this point it's probably easiest for me to just delete the incorrect code, instead of trying to get ChatGPT to produce better output.

I don't love this code, but it looks otherwise functional.

This process kind of felt like asking a junior programmer to write me some code, but ChatGPT wrote it faster.

As a DevOps engineer I end up writing short scripts in various languages like this fairly often, so a productivity boost would be helpful. Time-wise, it's not a huge part of my job, so more efficient AI powered script writing seems like an incremental improvement but it doesn't feel like something that would eliminate my job. I honestly am not an expert in bash, I end up doing a web search for the basics every time. I probably should just print a cheat sheet, but maybe ChatGPT could help.

ChatGPT doesn't feel productive to me in it's current state. The output takes longer to generate than a DDG search and result page takes to load. It seems reasonable to assume that ChatGPT can scale further and performance can improve, so those are "wait and see" problems.

I decided to work with ChatGPT some more to see if I could engineer the prompts to get working code that did what I needed.

I need a bash script I can put into a cron job. The script should take command line arguments for a domain name and an IP address. When run, the script should lookup the A record for the domain and exit 1 if the IP address doesn't match the command line argument.

[snip]

Instead of a command line argument, can you read the IP address from a file and save the lookup result to the same file at the end?

[snip]

Can you only have the IP address to the file if the dig command exited successfully?

[snip]

Instead of a command line argument, can you read the IP address from a file and save the lookup result to the same file at the end?

[snip]

Can you only have the IP address to the file if the dig command exited successfully?

[snip]

Can you only output text on an error?

[snip]

#!/bin/bash

# Parse command line argument for domain name
if [ $# -ne 1 ]; then
  echo "Usage: $0 domain_name"
  exit 1
fi

domain_name=$1
ip_file="$domain_name.ip"

# Read the expected IP address from the file
if [ -f "$ip_file" ]; then
  expected_ip=$(cat "$ip_file")
else
  echo "IP file not found: $ip_file"
  exit 1
fi

# Lookup the A record for the domain and sort the IP addresses
dns_result=$(dig +short $domain_name A | sort)

# Check the dig exit code before saving the result to the file
if [ $? -eq 0 ]; then
  echo "$dns_result" > "$ip_file"
else
  exit 1
fi

# Sort the expected IP address in case there are multiple addresses
expected_ip_sorted=$(echo "$expected_ip" | tr ' ' '\n' | sort | tr '\n' ' ')

# Sort the DNS result
dns_result_sorted=$(echo "$dns_result" | tr ' ' '\n' | sort | tr '\n' ' ')

# Check if the IP address matches the expected value
if [ "$dns_result_sorted" != "$expected_ip_sorted" ]; then
  exit 1
fi

[snip]

That took a couple rounds, but it worked. The problems took some analysis:

There's a minor oddity that it sorts the file contents every time. I'd prefer to sort the dig results before saving to the file, avoiding redundant sorts. This isn't a huge issue, the file is small and the sort will run quickly, but it feels odd.

I ended up pressing further to see if I could apply some best practices like using set -e at the start. This didn't go well.

Can you set -e at the start?

[snip]

Can you check the exit code of dig?

[snip]

#!/bin/bash
set -e

# Parse command line argument for domain name
if [ $# -ne 1 ]; then
  echo "Usage: $0 domain_name"
  exit 1
fi

domain_name=$1
ip_file="$domain_name.ip"

# Read the expected IP address from the file
if [ -f "$ip_file" ]; then
  expected_ip=$(cat "$ip_file")
else
  echo "IP file not found: $ip_file"
  exit 1
fi

# Lookup the A record for the domain and sort the IP addresses
dns_result=$(dig +short $domain_name A | sort)

# Check the dig exit code before saving the result to the file
if [ $? -eq 0 ]; then
  echo "$dns_result" > "$ip_file"
else
  exit 1
fi

# Sort the expected IP address in case there are multiple addresses
expected_ip_sorted=$(echo "$expected_ip" | tr ' ' '\n' | sort | tr '\n' ' ')

# Sort the DNS result
dns_result_sorted=$(echo "$dns_result" | tr ' ' '\n' | sort | tr '\n' ' ')

# Check if the IP address matches the expected value
if [ "$dns_result_sorted" != "$expected_ip_sorted" ]; then
  exit 1
fi

ChatGPT doesn't know that the script exited when dig failed. Does chat GPT know about this issue?

When using set -e how do I check the exit code of a command in bash?

[snip]

#!/bin/bash
set -e

# Run a command and check the exit code
ls /nonexistent
if [ $? -ne 0 ]; then
  echo "ls failed with exit code $?"
fi

Regenerate

[snip]

#!/bin/bash
set -e

# Run a command and check the exit code
some_command || {
  echo "The command failed with exit code $?"
  exit 1
}

ChatGPT got this the second time. Regenerate seems helpful, but you'd need to independently verify that there was a problem to know I needed to regenerate. But since I know this stuff already and this info is on StackOverflow, it's hard to see the usefulness of ChatGPT for something like this.

Also, when working with larger code samples, regenerate will produce wildly different code for the entire sample. So you can't just diff what was changed, you're starting from scratch.

Simple web games

Maybe ChatGPT doesn't know bash very well. I've seen people claim that they are able to use ChatGPT to program in unfamiliar languages and domains. Web front-end programming feels like a weakness to me, but let's see how far I can get.

At the start of the pandemic I wrote a couple short web games for my kids. The goal was pretty simple, make something that let a toddler gain familiarity with a computer. I built about six, aiming to build each in about half an hour, although the first one took longer. Sometimes, I had my Kindergartener sit with me. I talked through what changes I was making and solicited input on the content.

Let's start with the most basic one Put out the fire which teaches the user how to use the mouse.

Can you help me build a simple web-based game?

Of course, I'd be happy to help! What kind of game are you looking to create?

OK, first I need an 8x8 grid. Most of the cells should contain nothing, but between 2 and 30 of them should contain the fire emoji. Once the game is setup, the user should move their mouse across the screen. When the mouse is over a cell it should replace the cell with the water drop emoji. Once all the fire emoji cells are gone, the game is won. On winning, all cells of the game should be replaced with a fireman or firewoman emoji for two seconds. Then the game should reset so it can be played again.

Sure, I can help you create this game! Here is a basic outline of how you can create it:

Create an HTML file with an 8x8 grid. You can do this using the <table> element in HTML, and adding rows and columns using the <tr> and <td> elements.

Add the fire and water drop emojis to the appropriate cells. You can use the <img> element in HTML to add images to the cells.

Use JavaScript to randomly select between 2 and 30 cells to contain the fire emoji. You can use the Math.random() function to generate a random number between 2 and 30, and then loop through the cells in the grid to randomly select cells to contain the fire emoji.

Use JavaScript to detect when the user moves their mouse over a cell, and replace the cell with the water drop emoji. You can use the addEventListener() function in JavaScript to listen for the mouseover event, and then replace the image in the cell with the water drop emoji when the event is triggered.

Use JavaScript to detect when all the fire emoji cells have been replaced with water drop emojis, and display the fireman or firewoman emoji for two seconds before resetting the game. You can keep track of how many fire emoji cells are left, and when that number reaches 0, display the fireman or firewoman emoji using the setTimeout() function, and then reset the game by refreshing the page.

Here is some sample code to get you started:

HTML:

<table>
  <tr>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
    <td><img src="blank.png"></td>
  </tr>
  <!-- repeat this row 7 more times -->
</table>
JavaScript:
// get all the cells in the grid
const cells = document.getElementsByTagName("td");

// randomly select between 2 and 30 cells to contain the fire emoji
const numFire = Math.floor(Math.random() * 29) + 2;
const fireIndexes = [];
for (let i = 0; i < numFire; i++) {
  let index = Math.floor(Math.random() * 64);
  while (fireIndexes.includes(index)) {
    index = Math.floor(Math.random() * 64);
  }
  fireIndexes.push(index);
}

// add the fire and water drop emojis to the appropriate cells
for (let i = 0; i < cells.length; i++) {
  if (fireIndexes.includes(i)) {
    cells[i].innerHTML = '<img src="fire.png">';
  } else {
    cells[i].innerHTML = '<img src="blank.png">';
  }
}

// keep track of how many fire emoji cells are left
let numFireLeft = numFire;

// listen for the mouseover event on each cell
for (let i = 0; i < cells.length; i++) {
  cells[i].addEventListener("mouseover", function() {
    if (this.innerHTML === '<img src="fire.png">') {
      this.innerHTML = '<img src="water.png">';
      numFireLeft--;
      if (numFireLeft === 0) {
        // display the fireman or firewoman emoji for two seconds
        setTimeout(function() {
          for (let j = 0; j < cells.length; j++) {
            cells[j].
            

It's hard to evaluate this as the code just stops. I guess that's the response limit?

Can you complete the JavaScript code?

Can you use "๐Ÿ”ฅ" as the fire emoji, "๐Ÿ’ง" as the water emoji, "๐Ÿ‘จโ€๐Ÿš’" as the fireman emoji, and "๐Ÿ‘ฉโ€๐Ÿš’" as the firewoman emoji?

[snip]

// get all the cells in the grid
const cells = document.getElementsByTagName("td");

// randomly select between 2 and 30 cells to contain the fire emoji
const numFire = Math.floor(Math.random() * 29) + 2;
const fireIndexes = [];
for (let i = 0; i < numFire; i++) {
  let index = Math.floor(Math.random() * 64);
  while (fireIndexes.includes(index)) {
    index = Math.floor(Math.random() * 64);
  }
  fireIndexes.push(index);
}

// add the fire and water drop emojis to the appropriate cells
for (let i = 0; i < cells.length; i++) {
  if (fireIndexes.includes(i)) {
    cells[i].innerHTML = '๐Ÿ”ฅ';
  } else {
    cells[i].innerHTML = '';
  }
}

// keep track of how many fire emoji cells are left
let numFireLeft = numFire;

// listen for the mouseover event on each cell
for (let i = 0; i < cells.length; i++) {
  cells[i].addEventListener("mouseover", function() {
    if (this.innerHTML === '๐Ÿ”ฅ') {
      this.innerHTML = '๐Ÿ’ง';
      numFireLeft--;
      if (numFireLeft === 0) {
        // display the fireman or firewoman emoji for two seconds
        setTimeout(function() {
          for (let j = 0; j < cells.length; j++) {
            if (Math.random() < 0.5) {
              cells[j].innerHTML = '๐Ÿ‘จโ€๐Ÿš’';
            } else {
              cells[j].innerHTML = '๐Ÿ‘ฉโ€๐Ÿš’';
            }
          }
          setTimeout(function() {
            location.reload();
          }, 2000);
        }, 500);
      }
    }
  });
}

A quick pass over the logic, this all seems OK. However, if we simply paste the HTML into a file and added a <script> element for the JavaScript here's what we'd see.

You may see something different, but I'm seeing a game where you've got to replace ยฅ with ยง. From domain knowledge, you need to start your HTML file in order to use unicode. ChatGPT only gave us a snippet.

OK, so we'll need to ask ChatGPT for more help on setting up out HTML file.

What should the HTML file look like?

[snip]

<!DOCTYPE html>
<html>
  <head>
    <title>Firefighter Game</title>
    <meta charset="UTF-8">
    <style>
      table {
        border-collapse: collapse;
      }
      td {
        width: 50px;
        height: 50px;
        border: 1px solid black;
        text-align: center;
        font-size: 30px;
        cursor: pointer;
      }
    </style>
  </head>
  <body>
    <h1>Firefighter Game</h1>
    <table>
      <tbody>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
        <tr>
          <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
        </tr>
      </tbody>
    </table>
    <script src="game.js"></script>
  </body>
</html>
            

OK, this is a playable game!

There's some tweaks I'd want to do, but this was actually quite quick to produce. It was faster than it originally took for me to write.

The style isn't what I wanted though. The cells are too small, I want oversized cells that fill the screen. Water should also spray onto the blank cells; this honestly seems like the appeal of the game, I'd see the kids spray the entire board, saving one fire cell for last so they could get everything wet.

I wasn't able to make those changes. I asked for a couple style fixes, but it only centered the grid. The explainer text claimed the grid would fill the screen, but it didn't. When I pressed it to change the JavaScript, it crashed and was not able to continue the chat.

This felt much more impressive. I got a working game quickly. Other than asking for the full HTML to solve a charset issue, I didn't need to provide any domain knowledge. In terms of AI code generation, that was amazing and felt semi-accessible to non-domain experts.

I suspect part of my enthusiasm here is that I'm not thinking of production grade code (like I was for my bash code). If this was a commercial game, I'd want the game to reset in JavaScript instead of a page reload that would consume extra server-side resources and would require an active Internet connection. I wonder if this is why programmers claim ChatGPT helps them seem productive outside their traditional domain (they have a lower quality bar and are more willing to accept something that doesn't quite match what they wanted).

Simple React web games

OK, plain JavaScript isn't cool. What about React? Let's build the same game.

I'd like to build a simple game in react. Can you help?

Can you build a React game that has an 8x8 grid. Most of the cells should be empty, but between 2 and 30 of them should contain the fire emoji "๐Ÿ”ฅ". The user moves their mouse over the cells which changes them to the water drop emoji "๐Ÿ’ง". Once all the fire emoji cells have been converted, the game is won. The cells should all display a fire engine emoji "๐Ÿš’" for two seconds before the game resets.

[snip]

import React, { useState, useEffect } from "react";

const ROWS = 8;
const COLS = 8;
const NUM_FIRE_EMOJIS = Math.floor(Math.random() * 29) + 2; // Choose a random number of fire emojis between 2 and 30

function Game() {
  const [board, setBoard] = useState(createBoard());

  // Helper function to create the initial board with random fire emojis
  function createBoard() {
    const board = Array(ROWS).fill().map(() => Array(COLS).fill(" "));
    for (let i = 0; i < NUM_FIRE_EMOJIS; i++) {
      let row, col;
      do {
        row = Math.floor(Math.random() * ROWS);
        col = Math.floor(Math.random() * COLS);
      } while (board[row][col] === "๐Ÿ”ฅ");
      board[row][col] = "๐Ÿ”ฅ";
    }
    return board;
  }

  // Function to handle when the user moves their mouse over a cell
  function handleCellHover(row, col) {
    if (board[row][col] === "๐Ÿ”ฅ") {
      // If the cell contains fire, replace it with water
      const newBoard = [...board];
      newBoard[row][col] = "๐Ÿ’ง";
      setBoard(newBoard);

      // Check if all fire emojis have been converted to water
      if (newBoard.flat().every(cell => cell !== "๐Ÿ”ฅ")) {
        setTimeout(() => {
          setBoard(createBoard());
        }, 2000); // Wait 2 seconds before resetting the board
      }
    }
  }

  // Effect to reset the board when the component first mounts
  useEffect(() => {
    setBoard(createBoard());
  }, []);

  // Render the board
  return (
    <div className="board">
      {board.map((row, i) => (
        <div key={i} className="row">
          {row.map((cell, j) => (
            <div
              key={j}
              className="cell"
              onMouseEnter={() => handleCellHover(i, j)}
            >
              {cell === "๐Ÿ”ฅ" ? "๐Ÿš’" : cell}
            </div>
          ))}
        </div>
      ))}
    </div>
  );
}

export default Game;
            

I've trimmed out some of the ChatGPT helper text here. ChatGPT was fairly chatty, but it did helpfully link to a getting started guide. Unfortunately, that link has a banner saying that it's "no longer updated". The first page of ChatGPT mentions that it doesn't have much knowledge about things after 2021, maybe that deprecation was recent? I wonder how much impact that lag can cause, the JavaScript ecosystem seems to be very fast moving, being stale by two years could miss some important things.

It looks like there's some issues understanding the prompt, but the code can be run. I'll apply some of my domain knowledge to run create-react-app and drop the code in. I could ask ChatGPT, but it honestly doesn't feel productive...

When run, the code looks like this.

Right away, there's problems. There's no grid, there's no fire, and there's no winning animation.

At this point ChatGPT was crashing again, so I stopped here. ChatGPT clearly has stability problems.

Closing Thoughts

ChatGPT is overhyped. It could grow into something more, but it's a hot mess right now. My general stance is to "wait and see" how the tech develops and let the early adopters take the risk and pain of trying to get value out of this.

ChatGPT is overly confident in it's answers, and it was wrong more often than I expected, so you need to be careful with how you use it.

Does ChatGPT lower the barrier of entry for people without domain knowledge to program?

Yes, but only for a certain classes of problems. ChatGPT lacks domain knowledge so you can easily get stuck. ChatGPT can give you wrong information, which you'd need to independently understand and then correct. That's going to be very painful without domain knowledge.

There's still so much you need to fill in. I remember in CS 101, the instructor was careful to mention that you want to write code in a text editor, not MS Word or other "rich" editor. It felt like a basic thing, but it's one of a million things you've got to learn. You've got to install tools like NPM, you've got to navigate the command line, you've got to know how to test something locally, you need to know about building code, you need to know about deploying code.

Maybe ChatGPT can help you learn all of those things, if you know how to ask it. But now we're talking about someone who is gaining domain knowledge, instead of someone who doesn't need it. Can you learn to program faster using ChatGPT?

This smells like no-code

No-code marketing has been around for a while and their key claim is that they have found a way to let non-programmers do programming-like tasks without the expensive training and domain knowledge. I explored a couple of those tools once, but they all seems to focus on happy paths and used very leaky abstractions that required domain knowledge to fix. If you try simple examples, it's wonderful, but then you hit a snag and it's really ugly. ChatGPT's approach of generating code (which the non-programmer can't really read or understand) from a prompt and then using follow-up prompts to tweak the code looks very different, but seems subject to the same problems. At some point, you're going to fall off the happy path and you're going to need domain knowledge to fix the issue.

But the differences are interesting too. With ChatGPT you get code for a real programming language that an expert can debug, vs some XML generated mess that you really don't want to ever look at. Many no-code tools are closed and proprietary with almost zero "code" samples on GitHub, yet ChatGPT is currently open to the world to explore.

Each time I've been asked to look at no-code, I've steered projects away. I've seen other projects using no-code, but never successfully. So I'm biased to think ChatGPT would face a similar fate.

This feels like Wolfram Alpha again

When Wolfram Alpha first came out, I was stunned by how it worked. It seemed like we'd unlocked something truly special. Ask a question and it would churn out the answer. But it wouldn't always work either. At one point Siri could get information from Wolfram Alpha, but then they removed it.

I've probably used Wolfram Alpha a dozen times since an initial fascination. I just haven't found many times when it's useful and productive to use it. Presumably it's widely used in certain domains. Maybe ChatGPT will find a couple niches where it succeeds, but interest will fall in the others. It's possible that neither tool will kick-off a revolution.

Software Developer Jobs

I don't see any rapid changes occurring here. The tool isn't stable, accurate, or productive enough to make a domain expert faster. It's also not easy enough to use for a novice to crank out production level code for anything beyond toy projects. The remaining challenges are still huge.

The present risk seems limited to really small projects, where quality isn't important and the customer isn't picky if it's a little different than they wanted. I could see either a surge in development and/or a drop in the cost of small, low quality projects. But most customers will get picky, and want small changes to behavior or visual design; ChatGPT doesn't feel productive for fine-tuning.

Corporate Policy, Ownership

Before any employee starts slinging code into ChatGPT, a company lawyer really needs to take a look. Are you granting a license to ChatGPT to incorporate your code into their language model / training set?

OpenAI isn't actually open. As you work with ChatGPT you are helping it train and improve. Why are you contributing free labor to a closed source product? OpenAI Inc. may be non-profit, but it's subsidiary OpenAI LP is for-profit.

Are you training an AI to do your job for you, while funneling future profits from your domain expertise to the OpenAI investors? I'm extremely wary of who's in control of the language model and how access may be limited and monetized in the future.






Hello! I'm Robert Alexander, a freelance software developer and security consultant. This blog features some of my work and thoughts on software, the cloud, and security. You can get updates on my posts with your favorite RSS client or on Substack. I'm also on Mastodon.