Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!
  • Guest, before posting your code please take these rules into consideration:
    • It is required to use our BBCode feature to display your code. While within the editor click < / > or >_ and place your code within the BB Code prompt. This helps others with finding a solution by making it easier to read and easier to copy.
    • You can also use markdown to share your code. When using markdown your code will be automatically converted to BBCode. For help with markdown check out the markdown guide.
    • Don't share a wall of code. All we want is the problem area, the code related to your issue.


    To learn more about how to use our BBCode feature, please click here.

    Thank you, Code Forum.

Node.JS How can I restructure my loop to paginate faster right now it takes too long?

odnil

New Coder
This takes ages to retrieve data. Is there a more efficient way to do this?
Per_page limit is 100 & I needed the function to be able to find results for any date range using pagination. So I added a do while loop and this is what I ended up with:

JavaScript:
async function foo(repoOwner, repository, startingDate, endingDate){
    const listOfUserObjects = [];
    let pages, page = 1;
    do{
        await axios({
            method: 'get',
            url : `https://api.github.com/repos/${repoOwner}/${repository}/pulls?state=all&per_page=100&page=${page}`,
        }).then(response => {
            const users = response.data, startAt = new Date(startingDate), endAt = new Date(endingDate.valueOf());
            endAt.setDate(endAt.getDate() + 1);
            if(page === 1) pages = (Math.floor((users[0].number)/100))/2, console.log("collecting data, please wait...");
            for(let i = 0; i < users.length; i++) {
                const createdDate = new Date(users[i].created_at), updatedDate = new Date(users[i].updated_at), closedDate = new Date(users[i].closed_at), mergedDate = new Date(users[i].merged_at);
                if(((createdDate || updatedDate || closedDate || mergedDate) >= startAt) && ((createdDate || updatedDate || closedDate || mergedDate) <= endAt)) {
                    const userObj = {};
                    userObj.id = users[i].id;
                    userObj.user = users[i].user.login;
                    userObj.title = users[i].title;
                    userObj.state = users[i].state;
                    userObj.created_at = users[i].created_at.slice(0,10);
                    listOfUserObjects.push(userObj);
                } else {
                    continue;
                };
            }
            page++
        }).catch(error => {
            throw error.response.status === 404 ? Error("Error 404 User or Repo Not Found") : Error(error);
        });
    } while(page < pages);
    console.log(listOfUserObjects);
}
 
This takes ages to retrieve data. Is there a more efficient way to do this?
Per_page limit is 100 & I needed the function to be able to find results for any date range using pagination. So I added a do while loop and this is what I ended up with:

JavaScript:
async function foo(repoOwner, repository, startingDate, endingDate){
    const listOfUserObjects = [];
    let pages, page = 1;
    do{
        await axios({
            method: 'get',
            url : `https://api.github.com/repos/${repoOwner}/${repository}/pulls?state=all&per_page=100&page=${page}`,
        }).then(response => {
            const users = response.data, startAt = new Date(startingDate), endAt = new Date(endingDate.valueOf());
            endAt.setDate(endAt.getDate() + 1);
            if(page === 1) pages = (Math.floor((users[0].number)/100))/2, console.log("collecting data, please wait...");
            for(let i = 0; i < users.length; i++) {
                const createdDate = new Date(users[i].created_at), updatedDate = new Date(users[i].updated_at), closedDate = new Date(users[i].closed_at), mergedDate = new Date(users[i].merged_at);
                if(((createdDate || updatedDate || closedDate || mergedDate) >= startAt) && ((createdDate || updatedDate || closedDate || mergedDate) <= endAt)) {
                    const userObj = {};
                    userObj.id = users[i].id;
                    userObj.user = users[i].user.login;
                    userObj.title = users[i].title;
                    userObj.state = users[i].state;
                    userObj.created_at = users[i].created_at.slice(0,10);
                    listOfUserObjects.push(userObj);
                } else {
                    continue;
                };
            }
            page++
        }).catch(error => {
            throw error.response.status === 404 ? Error("Error 404 User or Repo Not Found") : Error(error);
        });
    } while(page < pages);
    console.log(listOfUserObjects);
}
Hi there, first of all, is there a particular reason why you're iterating through github repos?
 
Hi there, first of all, is there a particular reason why you're iterating through github repos?
Hi Antero, any assistance will be appreciated. I'm currently a student & this is one of my exercises, I'm guessing the point is for me to learn to interact with api's & use pagination. I'm also not allowed to use the github search api in my URL.
 
I tested your code against the axios / axios repo, and yes it does take a long time to get all pages. I'm not at all sure you could get that much faster. Actually what you are doing is not pagination, you are just getting all the pages in one go (albeit in batches of 100). The idea of pagination, I believe, is to get and display the first 100 pages, and only get the next 100 when the user asks for the next page. I would assume that is what the exercise expects you to program. Display 100, and have a Prev and Next button to scroll up and down.

Some points about your code:

1) I do not understand the line pages = (Math.floor((users[0].number)/100))/2; Why the division by two ?
2) Are you sure this condition if(((createdDate || updatedDate || closedDate || mergedDate) >= startAt) works as you expect ? In all languages I know (except Algol, which did allow this kind of logic) this should be if((createdDate>= startAt || updatedDate>= startAt || closedDate>= startAt || mergedDate >= startAt). Seeing JS does not throw an error, it must be doing something, but I can only guess what exactly...
3) Line 29 is missing a semicolon. Not really an error, but do be consistent.
 
Hi there, first of all, is there a particular reason why you're iterating through github repos?
Hi Antero, any assistance will be a. I'm currently a student & this is one of my exercises, I'm guessing the point is for me to learn to interact with api's & use pagination. I'm also not allowed to use the github search api in my URL.
Can you please share the parameters that you are passing to function foo() ?
Hi cbreemer. Should be able to process any public repo. If no pull requests for date range exist an empty array is printed out.
e.g. foo("<repo owner>", "<repo name>", "<YYYY-MM-DD>", "<YYYY-MM-DD>");
 
I tested your code against the axios / axios repo, and yes it does take a long time to get all pages. I'm not at all sure you could get that much faster. Actually what you are doing is not pagination, you are just getting all the pages in one go (albeit in batches of 100). The idea of pagination, I believe, is to get and display the first 100 pages, and only get the next 100 when the user asks for the next page. I would assume that is what the exercise expects you to program. Display 100, and have a Prev and Next button to scroll up and down.

Some points about your code:

1) I do not understand the line pages = (Math.floor((users[0].number)/100))/2; Why the division by two ?
2) Are you sure this condition if(((createdDate || updatedDate || closedDate || mergedDate) >= startAt) works as you expect ? In all languages I know (except Algol, which did allow this kind of logic) this should be if((createdDate>= startAt || updatedDate>= startAt || closedDate>= startAt || mergedDate >= startAt). Seeing JS does not throw an error, it must be doing something, but I can only guess what exactly...
3) Line 29 is missing a semicolon. Not really an error, but do be consistent.
1. Getting roughly half of all the pulls was the intent on my part, hence the division by 2 (mistakenly thought that this was pagination).
2. I will troubleshoot this to be certain, but so far it returned the data I expected (although the time taken is too long)
3. noted.
I've also tried getting rid of the loop altogether and instead altered my URL to this: https://api.github.com/search/issues?q=repo:${owner}/${repo}+is: pr+created:${Date}..${endDate}
this was faster as expected, but the ID did not match what was expected, so I reverted back to what I have now, and thus back to the speed problem.
 
Last edited:
1. Getting roughly half of all the pulls was the intent on my part, hence the division by 2 (mistakenly thought that this was pagination).
Aha. And here was me thinking it was something really deep that I was missing 😆
2. I will troubleshoot this to be certain, but so far it returned the data I expected (although the time taken is too long)
Hmmm, if you say so.... I just can't believe that JS will juggle these dates in the way you intended but perhaps one of the real JS experts here can chip in ?
Anyway, I would recommend to write it the way I suggested, just to make sure it will work, and to avoid any unnecessary discussion when you hand in your code.

I've also tried getting rid of the loop altogether and instead altered my URL to this: https://api.github.com/search/issues?q=repo:${owner}/${repo}+is: pr+created:${Date}..${endDate}
this was faster as expected, but the ID did not match what was expected, so I reverted back to what I have now, and thus back to the speed problem.
Yep, getting all the data with one GET request will be faster than using multiple requests, plus that your date comparison is done on the server. I don't get your point about the ID not matching though ... not matching what ? Anyway, it seems your exercise requires pagination, which this isn't either.
 

New Threads

Latest posts

Buy us a coffee!

Back
Top Bottom