Handling Proxy Authentication in node-fetch

The idea of proxy authentication in node-fetch is simple: Suppose that you have to build workflows for web scraping or make API requests on behalf of a user. Node-fetch does not have built-in proxy support, but it can use proxy servers and authentication dynamically.

Why Proxy Authentication Is Required in node-fetch

The authenticating proxy is only going to proxy certain folks through the server. If your proxy isn't setup to authenticate proxy requests then it's simply not going to accept proxy requests; instead you'll get a 407 or just silently not work. Properly credentialing your proxy is the cornerstone of any secure proxy.

Setting Up Proxy Authentication with https-proxy-agent

Step 1: Get the Package

Node-fetch does not support proxy. You need to use a module called https-proxy-agent to configure an authenticated proxy.

npm install node-fetch@2 https-proxy-agent

Step 2: Forward Credentials in Proxy URL

The simplest way is to put the credentials in the proxy string (just don't share this code with anyone!):

const fetch = require('node-fetch');
const { HttpsProxyAgent } = require('https-proxy-agent');

const proxyAgent = new HttpsProxyAgent(
  'http://username:password@proxy_host:port'
);

fetch('https://example.com', { agent: proxyAgent })
  .then(res => res.text())
  .then(console.log);

Step 3. Secure it with environment variables

Don't hardcode your login data into source code. Save your login data to a.env file then verify the info using process.env:

const proxyUrl = `http://${process.env.PROXY_USER}:${process.env.PROXY_PASS}@proxy_host:port`;
const agent = new HttpsProxyAgent(proxyUrl);

fetch('https://example.com', { agent })
  .then(res => res.text())
  .then(console.log);

Tip: URL-Encode Special Characters in Credentials

All username or password with special characters like @,:,# etc. should be URL encoded before putting that value in the proxy string.

Frequent Authentication Problems and How to Remedy Them

Incorrect Credential Arrangement

Most common cause of auth failure is Invalid proxy URL. Make sure there is no typo in your.env file, no additional space and also port number.

Using SOCKS5 proxy

For SOCKS5 authenticated proxies: Use socks-proxy-agent instead of https-proxy-agent:

npm install socks-proxy-agent

const { SocksProxyAgent } = require('socks-proxy-agent');
const agent = new SocksProxyAgent('socks5://username:password@proxy_host:port');

Secure Proxy Authentication - Some Guidelines

Never hardcode credentials – always use an environment variable.
URL-encode special characters in usernames and passwords.
Best way to deflect rate limiting or IP ban. Use rotating proxies.
Add error handling to handle 407 auth errors gracefully.
Click here to read the full process of setup proxy. If you want to know about the entire process of setting a proxy, go through our blog on proxy setup.
Not sure what proxies are? Find out about proxy servers.

If you have any questions Contact Us

FAQs you Have To Know

Q1: How can I add proxy authentication to node-fetch?
Put your login in the proxy URL as http://username:password@host:port with HttpsProxyAgent.

Q2. How can I protect the proxy credentials?
Always use environment variable to store the proxy credentials, don't store it in the source code it has to be stored using.env file.

Q3: Manage proxy auth in node-fetch package?
Proxy authentication with node-fetch is done with the npm module https-proxy-agent.

Q4: How come proxy authentication is not working for me?
Verify the input for any typographical errors, Special characters are url encoded (if needed). Proxy server has been assigned the proper host/port.

Q5: Is it possible to do SOCKS5 proxy with authenticatio with node-fetch?
Yes, replace https-proxy-agent by socks-proxy-agent to your code to enable SOCKS5 support.

Handling Proxy Authentication in node-fetch: Tips and Tricks