Fetch all data from paginated api

Backstory, I want to fetch all the orders from shopify api. The problem is shopify only allows max 250 numbers of data per request. So, how do we get the list of all data when number of data is more than 250 ?

Here is how we I did it,

const axios = require('axios');

const username = 'xxx-7f24a64c482b';
const password = 'xxx-6632020936819';
const store= 'my-store'
const limit = 250;
async function getOrders() {
 const response = await axios.get(`https://${username}:${password}@${store}.myshopify.com/admin/api/2020-01/orders.json?limit=${limit}`)
 return response.data.orders
}

getOrders().then(response => console.log(response.data));

Here the getOrders sends the GET request to shopify to get the list of data, here the limit is 250 which is max, so we get only 250 data. NOTE If you don’t have more than 250 orders, then this should be enough, but we are looking for more than 250 so lets modify the function above,

async function getAllOrders(
  url = `https://${username}:${password}@${store}.myshopify.com/admin/api/2020-01/orders.json?limit=${limit}`
) {
  const response = await axios({
    method: 'get',
    url,
    headers: {
      'X-Shopify-Access-Token': 'xxx-9368196b',
    },
  });
  const { orders } = response.data;
}

getAllOrders function above sends the GET request to shopify store. Here we have passed url as an argument and pass that url to the axios. If the axios get request seems wierd, you can check the axios docs.

In Shopify, when you have data in the next page then you will get a link in a response header, which looks like this,

link: "<https://{shop}.myshopify.com/admin/api/{version}/products.json?page_info={page_info}&limit={limit}>; rel={next}, <https://{shop}.myshopify.com/admin/api/{version}/products.json?page_info={page_info}&limit={limit}>; rel={previous}"

If you see the link above it has < and >, so we need to remove those to get a valid url, we do this by using replace function.

const nextUrl = response.headers.link.replace(/<|>/g, '')

Now, we have the valid url that gets us the data from the next page, so where do we use it? If we look at our current function it uses the url that we pass as arguement to fetch the data. So, if we pass our new url to the getAllOrders function, that should do it right ? Yes, and it’s called recursion.

A recursive function is a function that calls itself during its execution. This enables the function to repeat itself several times, outputting the result and the end of each iteration.

So, lets make our current function a recursive.

const moreOrders = await getAllOrders(nextUrl);

Here, we get moreOrders by calling the getAllOrders function and passing new url as argument. Now, the question is when do we stop this function? We stop it when there is no more next link.

if (orders.length < limit) return orders;

You must have a condition for a recursive function to stop, otherwise it’s will keep on forever.

So, tha’s it basically we now return all the orders by combining orders and moreOrders

return [...orders, ...moreOrers]

Now, lets combine all the code together,

async function getAllOrders(
  url = `https://${username}:${password}@${store}.myshopify.com/admin/api/2020-01/orders.json?limit=${limit}`
) {
  const response = await axios({
    method: 'get',
    url,
    headers: {
      'X-Shopify-Access-Token': 'xxx-9368196b',
    },
  });
  const { orders } = response.data;
  if (orders.length < limit) return orders;
  const nextUrl = response.headers.link.replace(/<|>/g, '');
  const moreOrders = await getAllOrders(nextUrl);
  return [...orders, ...moreOrders];
}

getAllOrders().then(response => console.log(response.data));