HTTP/HTTPS Requests via Invoke-WebRequest PowerShell Cmdlet

6K

The Invoke-WebRequest cmdlet is a command in PowerShell that allows you to send HTTP and HTTPS requests to web servers and retrieve the response. It is primarily used for web scraping, automating web tasks, and interacting with web-based APIs.

With Invoke-WebRequest, you can also maintain cookies and other session state between requests by using a WebRequestSession. This is useful when automating websites that require cookie-based authentication. It provides a way to interact with web pages and retrieve HTML or other data from them.

Invoke-WebRequest: The Basics

This cmdlet was introduced in Windows PowerShell 3.0. It has different aliases depending on the PowerShell edition you’re using.

In Windows PowerShell 5.1, Invoke-WebRequest has the aliases curl, iwr, and wget. In PowerShell 7+, only the iwr alias is available because curl and wget are mapped to native command-line tools.

The basic usage of Invoke-WebRequest is retrieving the webpage. The only required information is the URL.

Invoke-WebRequest -Uri <website URL>

Maintaining Cookies and Web Sessions

With the help of Invoke-WebRequest, you can maintain cookies between multiple HTTP requests by using a WebRequestSession. This is useful when a website creates an authentication session after login and subsequent requests must use the same cookies.

In order to create a web session while logging in, use the command:

$loginResponse = Invoke-WebRequest `
-Uri "https://example.com/login" `
-Method Post `
-Body @{
username = "admin"
password = "P@ssw0rd"
} `
-SessionVariable webSession

To reuse the same session for subsequent requests, use the following command:

$response = Invoke-WebRequest `
-Uri "https://example.com/admin" `
-WebSession $webSession

$response.StatusCode
$response.Content

The -SessionVariable parameter creates and stores a WebRequestSession object containing cookies and other session state returned by the server. The -WebSession parameter then reuses that session for subsequent requests.

You can inspect the cookies stored in the session with the Cookies property. Do this with the command below:

$webSession.Cookies.GetCookies("https://example.com/")

For example, I’ll download the https://theitbros.com website and save it to the $web variable.

In Windows PowerShell 5.1, you may need to add the -UseBasicParsing parameter on systems where Internet Explorer components are unavailable (note that this parameter is no longer required in PowerShell 7+):

# Windows PowerShell 5.1 only
$web = Invoke-WebRequest `
    -Uri https://theitbros.com `
    -UseBasicParsing

Note. If you’re connecting to legacy web servers that don’t support modern TLS versions, you may need to explicitly enable TLS 1.2 in Windows PowerShell 5.1 before sending the request:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

This method is only required for Windows PowerShell 5.1. Keep in mind that PowerShell 7+ uses modern TLS defaults and typically does not require this config.

Next, Let’s find out the resulting data type:

$web.GetType()

The response object type depends on the PowerShell version.

You can check the HTTP response status code returned by the web server using the StatusCode property:

$web.StatusCode

For example, a value of 200 means that the request was successful.

What properties does this object have?

$web | Get-Member -MemberType Properties

You can already guess by the properties listed in the above screenshot the type of data you can get from each property.

For example, the Content property contains the HTML code of the webpage. While the RawContent property also includes the HTML code, including the headers.

The Images and Links properties contain information about the images and links on the website.

Tip. You can use Invoke-WebRequest when you need access to the full HTTP response (headers, cookies, status code, HTML content, downloaded files, etc.). If you’re working with REST APIs that return JSON, Invoke-RestMethod is usually a better choice because it automatically converts the response into PowerShell objects.

Using Invoke-WebRequest with REST APIs

In modern enterprise environments, you can use Invoke-WebRequest to communicate with REST APIs. It allows you to send HTTP requests, add authentication headers, submit JSON payloads, and automate interactions with web services.

For example, the following command sends a GET request to an API endpoint using a Bearer authentication token:

$headers = @{
Authorization = "Bearer $token"
}

Invoke-WebRequest `
-Uri "https://api.contoso.com/v1/users" `
-Headers $headers

The -Headers parameter allows you to pass additional HTTP headers (such as authentication tokens, content types, and custom API headers).

You can also send data to an API endpoint using a POST request. Here is an example:

$body = @{
Name = "John Smith"
Department = "IT"
}

Invoke-WebRequest `
-Method POST `
-Uri "https://api.contoso.com/v1/users" `
-ContentType "application/json" `
-Body ($body | ConvertTo-Json)

Note. PowerShell 7.4 changed the default encoding used for request bodies from ASCII to UTF-8. This is important when sending JSON/other text that contains non-ASCII characters (such as accented letters, Cyrillic, or Asian characters). On older PowerShell 7 versions, explicitly encode the request body as UTF-8 when necessary to avoid corrupted characters.

In this example:

• Method POST — specifies the HTTP request method.
• ContentType — defines the format of the request body.
• Body — contains the JSON payload sent to the API.

Note. When working with REST APIs that return JSON data, the Invoke-RestMethod cmdlet is often a better choice because it automatically converts JSON responses into PowerShell objects.

Web Scraping

Web scraping refers to the automated process of extracting information or data from websites. It involves writing code to retrieve specific data from web pages and saving it in a structured format, such as a spreadsheet or a database.

Web scraping enables you to gather data from multiple sources quickly and efficiently without the need for manual copying and pasting.

Responsible Web Scraping

Before scraping a website, you need to check its robots.txt file and review the website’s terms of use. These may define which areas of the site should not be accessed by automated crawlers/impose other restrictions. You should also avoid sending requests too quickly/creating unnecessary load on the target server. When processing many pages, don’t forget to add a delay between requests and implement appropriate error handling and retry limits.

Here is an example:

$urls = @(
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3"
)

foreach ($url in $urls) {
try {
$response = Invoke-WebRequest `
-Uri $url `
-ErrorAction Stop

# Process the response here

Start-Sleep -Seconds 2
}
catch {
Write-Warning "Failed to retrieve $url : $($_.Exception.Message)"
}
}

The appropriate request interval depends on the website, the number of pages being requested, and any published crawling/API limits. For large-scale data collection, you should prefer an official API when one is available.

Using Invoke-WebRequest, or any other tool, to do web scraping is not a one-size fits all process. You need patience, apart from your code-analyzing skills.

Note. The following web-scraping examples using Invoke-WebRequest work as of this writing. However, it may eventually stop working when the target websites change.

Example: Extract the Article Title and Links

Suppose I want to get a list of all articles shown on the https://theitbros.com homepage.

The first thing I’ll do is run the following command to store the webpage in a variable:

$s = Invoke-WebRequest -Uri https://theitbros.com

Next, I’ll inspect which properties I can use:

$s.Links | Get-Member -MemberType Properties

According to the screenshot below, the properties are:

• href — contains the bare URL address.
• outerHTML — container the HTML hyperlink code.
• tagName — the href property HTML tag.

Now that I know the properties, I can simply run the below command to get all URLs.

$s.Links | Select-Object href

As you can see above, the href property lists all URLs found on the website, including links to non-articles. So I need to filter out some URLs that are not relevant.

When I analyzed the URLs, I realized that I needed to exclude the following:

• The website’s root URL — https://theitbros.com/
• URLs with these parts:
  • /about-the-authors/
  • /category/
  • /contact-us/
• HTML A tags that don’t start with — <a href=”https://theitbros.com

The below code translates that logic into a PowerShell filter using Where-Object.

$s.Links | 
Where-Object { 
$_.href -ne "https://theitbros.com/" -and 
$_.href -notlike "*/privacy-policy/*" -and 
$_.href -notlike "*/category/*" -and 
$_.href -notlike "*/author/*" -and 
$_.href -notlike "*/about-the-authors/*" -and 
$_.href -notlike "*/contact-us/*" -and 
$_.href -notlike "*/wp-content/*" -and 
$_.href -notlike "*/feed/*" -and 
$_.href -notlike "*/tag/*" -and 
$_.href -match '^https://(www\.)?theitbros\.com/'
}

When I ran the above command, I got the below result.

At this point, the result is more clear. But I only want to extract the article title and URL. The Links objects already contain the visible link text, so we can use the innerText property instead of parsing HTML manually. In this case, I need to extract the title in between the > and < characters. To extract the visible text from each matching HTML element, we’ll use the element’s innerText property.

Below is the updated code:

Our script mentioned below excludes non-content URLs (such as author pages, category and tag archives, the privacy policy and contact pages), WordPress system paths (such as wp-content, and feed URLs).

# Get TheITBros.com article links 
## Store the page in a variable 
$s = Invoke-WebRequest -Uri https://theitbros.com -Verbose 
## Filter links 
$s.Links | 
Where-Object { 
$_.href -ne "https://theitbros.com/" -and 
$_.href -notlike "*/privacy-policy/*" -and 
$_.href -notlike "*/category/*" -and 
$_.href -notlike "*/author/*" -and 
$_.href -notlike "*/about-the-authors/*" -and 
$_.href -notlike "*/contact-us/*" -and
$_.href -notlike "*/wp-content/*" -and 
$_.href -notlike "*/feed/*" -and 
$_.href -notlike "*/tag/*" -and  
$_.href -match '^https://(www\.)?theitbros\.com/'
} | Select-Object `
@{ 
n = 'Title'; e = { 
$_.innerText
} 
}, 
@{ 
n = 'URL'; e = { 
$_.href 
} 
}

The result is shown below.

That’s how you can use the Invoke-WebRequest cmdlet to perform web scraping.

Resolving Shortened URLs

Are you familiar with URL shorteners like TinyURL, Bitly, and ShortURL? Their purpose is simple — to shorten URLs.

For example, these short URLs resolve to — https://theitbros.com/how-to-restore-domain-controller-from-backup/.

• https://tinyurl.com/yckmmarn
• https://bit.ly/3OpOZrB
• https://shorturl.at/JRU27

The Invoke-WebRequest can help extract the original URL where these short URLs redirect.

First, let’s get run the following Invoke-WebRequest using one of the shortened URLs:

$shortUrl = 'https://tinyurl.com/yckmmarn' 
$result = Invoke-WebRequest -Uri $shortUrl -UseBasicParsing

On Windows PowerShell, you can extract the original URL from the BaseResponse.ResponseUri.AbsoluteUri property.

Keep in mind that by default, Invoke-WebRequest automatically follows HTTP redirects. If you need to inspect the redirect response and see the original Location header, you need to disable automatic redirection:

Invoke-WebRequest -Uri $shortUrl -MaximumRedirection 0

This allows you to view the HTTP 3xx response and the URL specified in the Location header.

Note. When automatic redirection is disabled, Invoke-WebRequest may throw an error for 3xx responses. You should use error handling/inspect the exception response to retrieve the Location header.

On PowerShell Core, the original URL is in the property:

BaseResponse.RequestMessage.RequestUri.AbsoluteUri

Handling errors in Invoke-WebRequest

When using Invoke-WebRequest in production scripts, we recommend you to handle network and HTTP errors gracefully. This helps your scripts continue running/display meaningful error messages when a request fails. Here is an example:

try {
$response = Invoke-WebRequest `
-Uri "https://example.com" `
-ErrorAction Stop

$response.StatusCode
}
catch {
Write-Warning "Request failed: $($_.Exception.Message)"
}

Using the -ErrorAction Stop parameter ensures that HTTP and network errors are treated as terminating exceptions that can be handled by the catch block. The same error-handling method is useful when working with internal HTTPS services that use self-signed/privately issued certificates.

Working with Self-Signed Certificates

Internal APIs, test servers, and other services in corporate environments may use self-signed certificates/certificates issued by an internal certification authority. If the certificate is not trusted by the client machine, Invoke-WebRequest may fail with a TLS certificate validation error.

In PowerShell 7+, you can use the -SkipCertificateCheck parameter to bypass certificate validation. Hete is an example:

$response = Invoke-WebRequest `
-Uri "https://internal-api.contoso.com" `
-SkipCertificateCheck

Warning. -SkipCertificateCheck disables TLS certificate validation. You should use it only for testing/controlled internal environments. Do not use it as a permanent solution for production systems!

For production environments, the preferred solution is to use a certificate issued by a trusted public/internal certification authority and make sure the client machine trusts the corresponding root and intermediate certificates.

Note that Windows PowerShell 5.1 does not provide the -SkipCertificateCheck parameter. A commonly used workaround is to override the .NET certificate validation callback:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

try {
$response = Invoke-WebRequest `
-Uri "https://internal-api.contoso.com" `
-ErrorAction Stop

$response.StatusCode
}
finally {
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
}

Warning. Note that this method disables certificate validation and you should use it only temporarily for testing. The callback can affect HTTPS requests made by the current PowerShell process. Do not use this method in production!

Note that you should not bypass certificate validation for production environments. Instead, you need to use a certificate issued by a trusted internal certification authority (CA) and make sure the client machine trusts the required root and intermediate CA certificates. This allows Invoke-WebRequest to validate the server certificate normally while maintaining TLS security.

Wrapping Up

In conclusion, the Invoke-WebRequest PowerShell cmdlet is a powerful tool that allows system administrators and PowerShell users to send HTTP and HTTPS requests programmatically. With its rich set of features and user-friendly syntax, it simplifies the process of interacting with web resources, retrieving data, and automating web-based tasks.

We explored the fundamental concepts of making HTTP and HTTPS requests using Invoke-WebRequest, starting with the basic syntax and parameters. By examining the response object, we gained insights into the server’s response status, headers, and content.

By harnessing the power of Invoke-WebRequest, system administrators and PowerShell users can streamline their workflows, enhance productivity, and unlock a world of possibilities in their PowerShell scripting endeavors.