Introduction
PowerShell has several methods of making requests to websites and examining their contents and the responses from web servers. This page was created to look at the various methods and what was being returned by using them.
Bots
There's nothing wrong with bots visiting most websites, it's not illegal, but the use some of the data collected is put to could be. The problems there's just so many of them. Even this site serves about 10 times the number of pages to bots compared to human visitors.
AWStats statistics for brisray.com, March 2025
DNS Records
For a website to exist and be found by its name, it must have a DNS record. DNS records have all sorts of uses, and one of them is to translate a website name to a computer's IP address. PowerShell's Resolve-DnsName cmdlet is the easiest way to check the DNS records of a website.
For this example, we do not need all the information that Resolve-DnsName can supply, just the website's IP address. As usual there is more than one way to test if a website has a DNS record and an IP address.
In the following examples, I used 5 sites. Two are well knowm, two are names I made up and one is this site.
$sites = 'google.com', 'aaamadeupyyy.xyz', 'microsoft.com', 'anyoldsite.io', 'brisray.com'
foreach ($site in $sites) {
$ipaddress = ''
$ipaddress = (Resolve-DnsName -Name $site -NoHostsFile -ErrorAction SilentlyContinue).ipaddress
if ($null -eq $ipaddress) {
$newtext = 'DNS IP address not found'
Write-Host $site - $newtext -ForegroundColor Red
}
else {
$newtext = 'DNS IP address found'
Write-Host $site - $newtext - $ipaddress -ForegroundColor Green
}
}The output of the above script is:
google.com - DNS IP address found - 2607:f8b0:4009:803::200e 142.250.191.238 aaamadeupyyy.xyz - DNS IP address not found microsoft.com - DNS IP address found - 2603:1030:b:3::152 2603:1030:20e:3::23c 2603:1030:c02:8::14 2603:1020:201:10::10f 2603:1010:3:3::5b 13.107.246.59 anyoldsite.io - DNS IP address not found brisray.com - DNS IP address found - 71.66.98.203
Some notes on the above script:
The -NoHostsFile parameter is used to ignore the Hosts file on my own computer
-ErrorAction SilentlyContinue - prevents the PowerShell error messages appearing when it cannot find the DNS records for the two domain names
There is a slightly different method of obtaining the same information:
$sites = 'google.com', 'aaamadeupyyy.xyz', 'microsoft.com', 'anyoldsite.io', 'brisray.com'
foreach ($site in $sites) {
$DnsEntry = ""
$DnsEntry = Resolve-DnsName -Name $site -NoHostsFile -ErrorAction SilentlyContinue | Select-Object ipAddress
If ($DnsEntry -match '@') {
$DnsEntry = "DNS record IP address found."
Write-Host $site - $DnsEntry -ForegroundColor Green
}
else {
$DnsEntry = "DNS record IP address not found!"
Write-Host $site - $DnsEntry -ForegroundColor Red
}
}The output of the above script is:
google.com - DNS record IP address found. aaamadeupyyy.xyz - DNS record IP address not found! microsoft.com - DNS record IP address found. anyoldsite.io - DNS record IP address not found! brisray.com - DNS record IP address found.
The output of the above script is:
The IP addresses are retuned from the cmdlet all start with @. The $DnsEntry -match '@' statement just looks for the character.
Both scripts can be run in PowerShell 5.1 Desktop and PowerShell 7.5 Core.
Information Returned by a Web Server
A lot of information can be obtained from a web server along with the actual page, and there are are different methods to retrieve it. Take a look at the output of these lines of script:
$site = 'https://example.com'
[net.WebRequest]::Create("$site") | Get-Member
Invoke-WebRequest $site | Get-Member
Invoke-RestMethod $site | Get-MemberIn the above code I used Get-Member cmdlet which gets the members, properties and methods, of objects. Any object can be piped to the Get-Member cmdlet.
The code returns:
TypeName: System.Net.HttpWebRequest
Name MemberType Definition
---- ---------- ----------
Abort Method void Abort()
AddRange Method void AddRange(int from, int to), void AddRange(long from, long to), void AddRange(int range), void AddRange(long range), void AddRang…
BeginGetRequestStream Method System.IAsyncResult BeginGetRequestStream(System.AsyncCallback callback, System.Object state)
BeginGetResponse Method System.IAsyncResult BeginGetResponse(System.AsyncCallback callback, System.Object state)
EndGetRequestStream Method System.IO.Stream EndGetRequestStream(System.IAsyncResult asyncResult, [ref] System.Net.TransportContext context), System.IO.Stream En…
EndGetResponse Method System.Net.WebResponse EndGetResponse(System.IAsyncResult asyncResult)
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetLifetimeService Method System.Object GetLifetimeService()
GetObjectData Method void ISerializable.GetObjectData(System.Runtime.Serialization.SerializationInfo info, System.Runtime.Serialization.StreamingContext c…
GetRequestStream Method System.IO.Stream GetRequestStream(), System.IO.Stream GetRequestStream([ref] System.Net.TransportContext context)
GetRequestStreamAsync Method System.Threading.Tasks.Task[System.IO.Stream] GetRequestStreamAsync()
GetResponse Method System.Net.WebResponse GetResponse()
GetResponseAsync Method System.Threading.Tasks.Task[System.Net.WebResponse] GetResponseAsync() GetType Method type GetType()
InitializeLifetimeService Method System.Object InitializeLifetimeService()
ToString Method string ToString()
Accept Property string Accept {get;set;}
Address Property uri Address {get;}
AllowAutoRedirect Property bool AllowAutoRedirect {get;set;}
AllowReadStreamBuffering Property bool AllowReadStreamBuffering {get;set;}
AllowWriteStreamBuffering Property bool AllowWriteStreamBuffering {get;set;}
AuthenticationLevel Property System.Net.Security.AuthenticationLevel AuthenticationLevel {get;set;}
AutomaticDecompression Property System.Net.DecompressionMethods AutomaticDecompression {get;set;}
CachePolicy Property System.Net.Cache.RequestCachePolicy CachePolicy {get;set;}
ClientCertificates Property System.Security.Cryptography.X509Certificates.X509CertificateCollection ClientCertificates {get;set;}
Connection Property string Connection {get;set;}
ConnectionGroupName Property string ConnectionGroupName {get;set;}
ContentLength Property long ContentLength {get;set;}
ContentType Property string ContentType {get;set;}
ContinueDelegate Property System.Net.HttpContinueDelegate ContinueDelegate {get;set;}
ContinueTimeout Property int ContinueTimeout {get;set;}
CookieContainer Property System.Net.CookieContainer CookieContainer {get;set;}
Credentials Property System.Net.ICredentials Credentials {get;set;}
Date Property datetime Date {get;set;}
Expect Property string Expect {get;set;}
HaveResponse Property bool HaveResponse {get;}
Headers Property System.Net.WebHeaderCollection Headers {get;set;}
Host Property string Host {get;set;}
IfModifiedSince Property datetime IfModifiedSince {get;set;}
ImpersonationLevel Property System.Security.Principal.TokenImpersonationLevel ImpersonationLevel {get;set;}
KeepAlive Property bool KeepAlive {get;set;}
MaximumAutomaticRedirections Property int MaximumAutomaticRedirections {get;set;}
MaximumResponseHeadersLength Property int MaximumResponseHeadersLength {get;set;}
MediaType Property string MediaType {get;set;}
Method Property string Method {get;set;}
Pipelined Property bool Pipelined {get;set;}
PreAuthenticate Property bool PreAuthenticate {get;set;}
ProtocolVersion Property version ProtocolVersion {get;set;}
Proxy Property System.Net.IWebProxy Proxy {get;set;}
ReadWriteTimeout Property int ReadWriteTimeout {get;set;}
Referer Property string Referer {get;set;}
RequestUri Property uri RequestUri {get;}
SendChunked Property bool SendChunked {get;set;}
ServerCertificateValidationCallback Property System.Net.Security.RemoteCertificateValidationCallback ServerCertificateValidationCallback {get;set;}
ServicePoint Property System.Net.ServicePoint ServicePoint {get;}
SupportsCookieContainer Property bool SupportsCookieContainer {get;}
Timeout Property int Timeout {get;set;}
TransferEncoding Property string TransferEncoding {get;set;}
UnsafeAuthenticatedConnectionSharing Property bool UnsafeAuthenticatedConnectionSharing {get;set;}
UseDefaultCredentials Property bool UseDefaultCredentials {get;set;}
UserAgent Property string UserAgent {get;set;}
TypeName: Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
BaseResponse Property System.Net.Http.HttpResponseMessage BaseResponse {get;set;}
Content Property string Content {get;}
Encoding Property System.Text.Encoding Encoding {get;}
Headers Property System.Collections.Generic.Dictionary[string,System.Collections.Generic.IEnumerable[string]] Headers {get;}
Images Property Microsoft.PowerShell.Commands.WebCmdletElementCollection Images {get;}
InputFields Property Microsoft.PowerShell.Commands.WebCmdletElementCollection InputFields {get;}
Links Property Microsoft.PowerShell.Commands.WebCmdletElementCollection Links {get;}
OutFile Property string OutFile {get;}
RawContent Property string RawContent {get;set;}
RawContentLength Property long RawContentLength {get;}
RawContentStream Property System.IO.MemoryStream RawContentStream {get;set;}
RelationLink Property System.Collections.Generic.Dictionary[string,string] RelationLink {get;}
StatusCode Property int StatusCode {get;}
StatusDescription Property string StatusDescription {get;}
TypeName: System.String
Name MemberType Definition
---- ---------- ----------
Clone Method System.Object Clone(), System.Object ICloneable.Clone()
CompareTo Method int CompareTo(System.Object value), int CompareTo(string strB), int IComparable.CompareTo(System.Object obj), int IComparable[string].Comp…
Contains Method bool Contains(string value), bool Contains(string value, System.StringComparison comparisonType), bool Contains(char value), bool Contains…
CopyTo Method void CopyTo(int sourceIndex, char[] destination, int destinationIndex, int count), void CopyTo(System.Span[char] destination)
EndsWith Method bool EndsWith(string value), bool EndsWith(string value, System.StringComparison comparisonType), bool EndsWith(string value, bool ignoreC…
EnumerateRunes Method System.Text.StringRuneEnumerator EnumerateRunes()
Equals Method bool Equals(System.Object obj), bool Equals(string value), bool Equals(string value, System.StringComparison comparisonType), bool IEquata…
GetEnumerator Method System.CharEnumerator GetEnumerator(), System.Collections.IEnumerator IEnumerable.GetEnumerator(), System.Collections.Generic.IEnumerator[…
GetHashCode Method int GetHashCode(), int GetHashCode(System.StringComparison comparisonType)
GetPinnableReference Method System.Char&, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e GetPinnableReference()
GetType Method type GetType()
GetTypeCode Method System.TypeCode GetTypeCode(), System.TypeCode IConvertible.GetTypeCode()
IndexOf Method int IndexOf(char value), int IndexOf(char value, int startIndex), int IndexOf(char value, System.StringComparison comparisonType), int Ind…
IndexOfAny Method int IndexOfAny(char[] anyOf), int IndexOfAny(char[] anyOf, int startIndex), int IndexOfAny(char[] anyOf, int startIndex, int count)
Insert Method string Insert(int startIndex, string value)
IsNormalized Method bool IsNormalized(), bool IsNormalized(System.Text.NormalizationForm normalizationForm)
LastIndexOf Method int LastIndexOf(char value), int LastIndexOf(char value, int startIndex), int LastIndexOf(char value, int startIndex, int count), int Last…
LastIndexOfAny Method int LastIndexOfAny(char[] anyOf), int LastIndexOfAny(char[] anyOf, int startIndex), int LastIndexOfAny(char[] anyOf, int startIndex, int c…
Normalize Method string Normalize(), string Normalize(System.Text.NormalizationForm normalizationForm)
PadLeft Method string PadLeft(int totalWidth), string PadLeft(int totalWidth, char paddingChar)
PadRight Method string PadRight(int totalWidth), string PadRight(int totalWidth, char paddingChar)
Remove Method string Remove(int startIndex, int count), string Remove(int startIndex)
Replace Method string Replace(string oldValue, string newValue, bool ignoreCase, cultureinfo culture), string Replace(string oldValue, string newValue, S…
ReplaceLineEndings Method string ReplaceLineEndings(), string ReplaceLineEndings(string replacementText)
Split Method string[] Split(char separator, System.StringSplitOptions options = System.StringSplitOptions.None), string[] Split(char separator, int cou…
StartsWith Method bool StartsWith(string value), bool StartsWith(string value, System.StringComparison comparisonType), bool StartsWith(string value, bool i…
Substring Method string Substring(int startIndex), string Substring(int startIndex, int length)
ToBoolean Method bool IConvertible.ToBoolean(System.IFormatProvider provider)
ToByte Method byte IConvertible.ToByte(System.IFormatProvider provider)
ToChar Method char IConvertible.ToChar(System.IFormatProvider provider)
ToCharArray Method char[] ToCharArray(), char[] ToCharArray(int startIndex, int length)
ToDateTime Method datetime IConvertible.ToDateTime(System.IFormatProvider provider)
ToDecimal Method decimal IConvertible.ToDecimal(System.IFormatProvider provider)
ToDouble Method double IConvertible.ToDouble(System.IFormatProvider provider)
ToInt16 Method short IConvertible.ToInt16(System.IFormatProvider provider)
ToInt32 Method int IConvertible.ToInt32(System.IFormatProvider provider)
ToInt64 Method long IConvertible.ToInt64(System.IFormatProvider provider)
ToLower Method string ToLower(), string ToLower(cultureinfo culture)
ToLowerInvariant Method string ToLowerInvariant()
ToSByte Method sbyte IConvertible.ToSByte(System.IFormatProvider provider)
ToSingle Method float IConvertible.ToSingle(System.IFormatProvider provider)
ToString Method string ToString(), string ToString(System.IFormatProvider provider), string IConvertible.ToString(System.IFormatProvider provider)
ToType Method System.Object IConvertible.ToType(type conversionType, System.IFormatProvider provider)
ToUInt16 Method ushort IConvertible.ToUInt16(System.IFormatProvider provider)
ToUInt32 Method uint IConvertible.ToUInt32(System.IFormatProvider provider)
ToUInt64 Method ulong IConvertible.ToUInt64(System.IFormatProvider provider)
ToUpper Method string ToUpper(), string ToUpper(cultureinfo culture)
ToUpperInvariant Method string ToUpperInvariant()
Trim Method string Trim(), string Trim(char trimChar), string Trim(Params char[] trimChars), string Trim(System.ReadOnlySpan[char] trimChars)
TrimEnd Method string TrimEnd(), string TrimEnd(char trimChar), string TrimEnd(Params char[] trimChars), string TrimEnd(System.ReadOnlySpan[char] trimCha…
TrimStart Method string TrimStart(), string TrimStart(char trimChar), string TrimStart(Params char[] trimChars), string TrimStart(System.ReadOnlySpan[char]…
TryCopyTo Method bool TryCopyTo(System.Span[char] destination)
Chars ParameterizedProperty char Chars(int index) {get;}
Length Property int Length {get;}
As can be seen, a lot of information can be obtained from a web site programmatically. Several of the objects have several proeprties but they are accessed in the same basic way with the dot notation. For example:
$variable = (Invoke-WebRequest -Uri $site -SkipHttpErrorCheck).object[.object]
Because of hte way they work, Invoke-RestMethod is better at dealing with XML and JSON results, while Invoke-WebRequest is better at dealing with straight HTML results. Invoke-RestMethod automatically converts data from JSON or XML sources into a PowerShell objects.
Server HTTP Status Response Codes
When someone requests a resource from a web server, among the information sent back from the server is a three digit response code which gives the status of the request. A successful request results in the response of 200 but another of the responses is the famous "404 - Resoure not found" code.
The codes are important because they give some idea of what is happening on the server. There is a website, httpstat.us, where all the codes can be tested for. The following script tests the pages for the response codes that are most commonly encountered.
The following code only works in PowerShell Core. If run in PowerShell 5.1 Desktop all the return codes return 0.
$codes = '200', '301', '302', '303', '307', '308', '400', '401', '403', '404', '500', '502', '503'
foreach ($code in $codes) {
$site = 'https://httpstat.us/' + $code
$StatusCode ="0"
$StatusCode = (Invoke-WebRequest -Uri $site -SkipHttpErrorCheck).StatusCode;
if($StatusCode -eq 200) { Write-Host -Object "$site is up with status 200" -ForegroundColor Green }
else { Write-Host -Object "$site gives the status code $($StatusCode)" -ForegroundColor Red }
}The output of the above script is:
https://httpstat.us/200 is up with status 200 https://httpstat.us/301 is up with status 200 https://httpstat.us/302 is up with status 200 https://httpstat.us/303 is up with status 200 https://httpstat.us/307 is up with status 200 https://httpstat.us/308 is up with status 200 https://httpstat.us/400 gives the status code 400 https://httpstat.us/401 gives the status code 401 https://httpstat.us/403 gives the status code 403 https://httpstat.us/404 gives the status code 404 https://httpstat.us/500 gives the status code 500 https://httpstat.us/502 gives the status code 502 https://httpstat.us/503 gives the status code 503
Some notes on the above script:
This only works in PowerShell Core because -SkipHttpErrorCheck was only introduced in PowerShell Core 7. What it does is make the Invoke-WebRequest cmdlet ignore HTTP error statuses and continue to process responses. Notice that the 3xx redirect repsonse codes all return 200 OK, successful HTTP status. This is because the sites redirects the page to one that exists.
Sources and Resources
httpstat.us - Site to test for the various HTTP status codes
Analysing HTTP Headers with PowerShell - Script Wizards
Get-Member - Microsoft Learn
HTTP requests with PowerShell's Invoke-WebRequest - David Hamann
HTTP response status codes - MDN Web Docs
HTTP Status Messages - W3Schools
Invoke-RestMethod - Microsoft Learn
Invoke-WebRequest - Microsoft Learn
Invoke-WebRequest or Invoke-RestMethod? - Truesec
List of HTTP status codes - Wikipedia
Powershell: Invoke-RestMethod vs Invoke-WebRequest - CSMA
Resolve-DnsName - Microsoft Learn
WebRequest Class - Microsoft Learn